Identification generation method, device, electronic device and storage medium
By acquiring and stitching the local mask features of the target image and combining global features for text prediction, the problem of ignoring local details in the prior art is solved, and the prediction accuracy of entity identification is improved.
Patent Information
- Application Number
- CN202411111303.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-14
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-08-14
AI Technical Summary
In the prior art, visual entity links ignore local details during image encoding, resulting in a decrease in the prediction accuracy of entity identification.
By obtaining the local mask in the target image, determining the query mask, extracting the mask visual features and position features, and stitching them into regional features, text prediction is performed based on the global features of the target image, and target entity identification is generated.
The prediction accuracy of target entity identification is improved, pixel-level visual information and position information of the image are captured, and the understanding of the global and detailed characteristics of the target image is enhanced.
Smart Images

Figure CN118629042B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of computer technology, and in particular to an identification generation method, device, electronic device and storage medium. Background Art
[0002] Visual entity linking refers to matching objects in an image with corresponding entities in a knowledge base. In related technologies, the image to be identified is usually encoded as a global image feature, and the query text describing the target object in the image to be identified is used as a visual cue. Then, the entity identifier corresponding to the image is predicted based on the global image feature and the text features of the query text. However, global image features usually ignore local details in the image, resulting in information loss, thereby reducing the prediction accuracy of entity identifiers. Summary of the invention
[0003] The following is a summary of the subject matter of the detailed description of the present disclosure. This summary is not intended to limit the scope of the claims.
[0004] The embodiments of the present disclosure provide an identification generation method, device, electronic device and storage medium, which can improve the prediction accuracy of the target entity identification.
[0005] In one aspect, an embodiment of the present disclosure provides a method for generating an identifier, comprising:
[0006] Acquire a local mask corresponding to each candidate object in the target image, and determine a query mask from the multiple local masks, wherein the query mask is used to indicate a target object selected from the multiple candidate objects;
[0007] Performing feature extraction based on the query mask and each of the local masks on the target image to obtain a plurality of mask visual features, and extracting mask position features corresponding to the query mask and each of the local masks;
[0008] splicing each of the mask visual features with the corresponding mask position features to obtain a plurality of regional features;
[0009] Extracting a first image feature of the target image, and splicing the first image feature and a plurality of the regional features to obtain a target splicing feature;
[0010] Text prediction is performed based on the target concatenation features to generate a target entity identifier of the target object.
[0011] On the other hand, the embodiment of the present disclosure further provides an identification generation device, including:
[0012] An acquisition module, configured to acquire a local mask corresponding to each candidate object in a target image, and determine a query mask from a plurality of the local masks, wherein the query mask is used to indicate a target object selected from the plurality of candidate objects;
[0013] A feature extraction module, used to perform feature extraction on the target image based on the query mask and each of the local masks, obtain a plurality of mask visual features, and extract mask position features corresponding to the query mask and each of the local masks;
[0014] A first splicing module, used for splicing each of the mask visual features with the corresponding mask position features to obtain a plurality of regional features;
[0015] A second splicing module is used to extract a first image feature of the target image, and splice the first image feature and a plurality of the regional features to obtain a target splicing feature;
[0016] A generation module is used to perform text prediction based on the target splicing features and generate a target entity identifier of the target object.
[0017] Furthermore, the second splicing module is specifically used for:
[0018] Determine the mask area of each of the local masks respectively, and splice the regional features corresponding to each of the local masks in order of the size of the mask area to obtain a first splicing feature;
[0019] The first image feature, the first stitching feature, and the region feature corresponding to the query mask are stitched together to obtain a target stitching feature.
[0020] Furthermore, the second splicing module is specifically used for:
[0021] Constructing a prompt text for prompting the first language model to generate an entity identifier;
[0022] The text feature of the prompt text is extracted, and the first image feature, the text feature, the first splicing feature and the region feature corresponding to the query mask are spliced to obtain a target splicing feature.
[0023] Furthermore, the feature extraction module is specifically used for:
[0024] Performing multi-level feature extraction on the target image to obtain multi-level visual features of the target image;
[0025] Based on the query mask and each of the local masks, respectively, mask pooling is performed on the multi-level visual features to obtain a plurality of multi-level pooled features;
[0026] Feature fusion is performed on each of the multi-level pooling features to obtain multiple mask visual features.
[0027] Furthermore, the feature extraction module is specifically used for:
[0028] For any of the multi-level pooling features, sub-features of each level in the multi-level pooling features are mapped respectively to obtain multiple intermediate features of the same dimension, and each of the intermediate features is subjected to feature fusion to obtain a fused feature;
[0029] Multi-layer perception processing is performed on each of the fused features to obtain multiple mask visual features.
[0030] Furthermore, the acquisition module is specifically used for:
[0031] Acquire a target image and a prompt mark of the target image, wherein the target image includes a plurality of candidate objects, and the prompt mark is used to indicate a target object selected from the plurality of candidate objects;
[0032] The target image is segmented to obtain local masks corresponding to the candidate objects, and a query mask is determined in each of the local masks based on the prompt mark.
[0033] Furthermore, the acquisition module is specifically used for:
[0034] When the hint mark is a mark point, determining a query mask in each of the local masks based on a positional relationship between the mark point and each of the local masks;
[0035] Alternatively, when the hint mark is a mark box, determining the query mask in each of the local masks based on a matching degree between the mark box and a mask boundary of each of the local masks;
[0036] Alternatively, when the hint mark is a marked area, a query mask is determined in each of the local masks based on a matching degree between the marked area and each of the local masks.
[0037] Furthermore, the target entity identifier is generated by the first language model, and the identifier generation device further includes a training module, which is specifically used for:
[0038] Acquire a sample image and a second mask corresponding to a sample object in the sample image, segment the sample image, and obtain a first mask corresponding to each visual object in the sample image, wherein the sample object is one of the multiple visual objects;
[0039] Extracting a second image feature of the sample image, performing feature extraction based on the second mask and each of the first masks on the sample image to obtain a plurality of sample visual features, and extracting sample position features corresponding to the second mask and each of the first masks;
[0040] splicing the second image feature, the sample visual feature, and the sample position feature and inputting them into the first language model for text prediction to generate a prediction probability distribution, wherein the prediction probability distribution is used to determine the entity identifier of the sample object;
[0041] Obtain a first entity identifier of a sample entity linked to the sample image, determine a model loss based on the predicted probability distribution and the first entity identifier, and train the first large language model based on the model loss, wherein the sample entity is used to indicate the sample object.
[0042] Furthermore, the sample image, the second mask and the first entity identifier are all obtained from a data set, and the training module is further used for:
[0043] Acquire multiple original images and query texts corresponding to each of the original images, and determine identification information corresponding to each of the original images according to each of the original images and the corresponding query texts, wherein the query texts are used to prompt identification of the object of interest in the corresponding original image;
[0044] Acquire a plurality of candidate entities, and determine, based on the respective identification information, link entities corresponding to the respective original images from the plurality of candidate entities;
[0045] Based on each of the original images and the corresponding query text, respectively determine a labeling mask for each of the original images, wherein the labeling mask is used to indicate the corresponding object of interest;
[0046] Each of the original images, the corresponding annotation mask, and the corresponding linked entity are associated and stored in the data set, wherein the sample image is sampled from multiple original images, the second mask is the annotation mask corresponding to the sample image, and the sample entity is the linked entity linked to the sample image.
[0047] Furthermore, the above training module is specifically used for:
[0048] Inputting each of the query texts into the second largest language model for text prediction to generate summary texts corresponding to each of the original images;
[0049] Performing object detection based on each of the original images and the corresponding summarized text, respectively, to generate an original bounding box corresponding to each of the original images, wherein the original bounding box is used to indicate the corresponding object of interest;
[0050] Each of the original images and the corresponding original bounding boxes are respectively input into a first mask generation model for mask prediction to generate a labeled mask corresponding to each of the original images.
[0051] Furthermore, the above training module is also used for:
[0052] Acquire a reference name text corresponding to each of the link entities, wherein the reference name text is used to indicate the name of the reference entity, and the level of the reference entity in the knowledge base is higher than the level of the link entity in the knowledge base;
[0053] Inputting each of the original images and the corresponding reference name text into a second mask generation model for mask prediction to generate a reference mask corresponding to each of the original images;
[0054] Determining the matching degree between each of the annotated masks and the corresponding reference mask to obtain the target matching degree corresponding to each of the annotated masks;
[0055] When the target matching degree is less than a preset matching degree threshold, the original image corresponding to the target matching degree is discarded.
[0056] Furthermore, the above training module is also used for:
[0057] Counting the number of connected regions in each of the annotation masks to obtain the number of regions corresponding to each of the annotation masks;
[0058] When the number of regions is greater than a preset number threshold, the original image corresponding to the number of regions is discarded.
[0059] Furthermore, the above training module is specifically used for:
[0060] Acquire a sample name text of a sample entity linked to the sample image, and segment the sample name text to obtain a plurality of first segmented words;
[0061] Determine the occurrence frequency of each of the first participles in the knowledge base, sort each of the first participles based on the order of the occurrence frequency from small to large, and determine the first participles arranged in the first L positions as the second participles, where L is a positive integer;
[0062] Based on each of the second participles, a first entity identifier of the sample entity is determined.
[0063] On the other hand, an embodiment of the present disclosure further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned identification generation method when executing the computer program.
[0064] On the other hand, an embodiment of the present disclosure further provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program is executed by a processor to implement the above-mentioned identification generation method.
[0065] On the other hand, the embodiment of the present disclosure further provides a computer program product, which includes a computer program, and the computer program is stored in a computer-readable storage medium. The processor of the computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the above-mentioned identification generation method.
[0066] The disclosed embodiments include at least the following beneficial effects: by acquiring the local mask corresponding to each candidate object in the target image, and determining the query mask used to indicate the target object, and then obtaining the query mask and the mask visual features corresponding to each local mask through feature extraction, and extracting the query mask and the mask position features corresponding to each local mask, the mask visual features can capture the pixel-level visual information of the local area where the corresponding object is located, the mask position features can capture the pixel-level position information of the local area where the corresponding object is located, and then splicing each mask visual feature with the corresponding mask position feature respectively to obtain the regional features of each local area, which is equivalent to combining the pixel-level visual information of the local area and the corresponding pixel-level position information into pixel-level regional information, and then extracting the target image. The first image feature of the image is captured, and the first image feature and multiple regional features are spliced into a target splicing feature. Then, text prediction is performed based on the target splicing feature to generate a target entity identifier of the target object. In the text prediction process, by interacting with each feature in the target splicing feature, it is possible to pay attention to both the global visual information captured by the first image feature and the pixel-level regional information corresponding to each candidate object and the pixel-level regional information corresponding to the target object. Therefore, it is possible to effectively improve the understanding of the global image features and pixel-level details of the target image, thereby improving the prediction accuracy of the target entity identifier. In addition, using the pixel-level query mask as a visual cue can efficiently, flexibly and accurately refer to the target object, thereby further improving the prediction accuracy of the target entity identifier.
[0067] Other features and advantages of the present disclosure will be set forth in the following description, and in part will be apparent from the description, or may be learned by practicing the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0068] The accompanying drawings are used to provide further understanding of the technical solution of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the technical solution of the present disclosure and do not constitute a limitation on the technical solution of the present disclosure.
[0069] Figure 1 A schematic diagram of an optional implementation environment provided for an embodiment of the present disclosure;
[0070] Figure 2 A schematic diagram of an optional flow chart of a method for generating an identifier provided in an embodiment of the present disclosure;
[0071] Figure 3 A schematic diagram of an optional process for determining visual features of a mask provided in an embodiment of the present disclosure;
[0072] Figure 4 A schematic diagram of an optional process of generating a mask provided in an embodiment of the present disclosure;
[0073] Figure 5 A schematic diagram of an optional process of updating a data set provided in an embodiment of the present disclosure;
[0074] Figure 6 A schematic diagram of an optional architecture for removing original images provided in an embodiment of the present disclosure;
[0075] Figure 7 An optional distribution diagram of entity categories in multiple sample sets provided by an embodiment of the present disclosure;
[0076] Figure 8 An optional pie-shaped schematic diagram of entity categories in the optimized sample set provided by an embodiment of the present disclosure;
[0077] Fig. 9 An optional distribution diagram of the area ratio of the annotation mask provided in the embodiment of the present disclosure;
[0078] Fig.10 A schematic diagram of an optional architecture of the training phase provided in the embodiment of the present disclosure;
[0079] Fig.11 A schematic diagram of an optional structure of the identification generating device provided in the embodiment of the present disclosure;
[0080] Fig.12 A partial structural block diagram of a terminal provided in an embodiment of the present disclosure;
[0081] Fig.13 A partial structural block diagram of a server provided in an embodiment of the present disclosure. DETAILED DESCRIPTION
[0082] In order to make the purpose, technical solution and advantages of the present disclosure more clear, the present disclosure is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present disclosure and are not used to limit the present disclosure.
[0083] It should be noted that in various specific embodiments of the present disclosure, when it comes to the need to perform relevant processing based on data related to the characteristics of the target object such as the target object attribute information or attribute information set, the permission or consent of the target object will be obtained first, and the collection, use and processing of these data will comply with relevant laws, regulations and standards. Among them, the target object can be a user. In addition, when the embodiment of the present disclosure needs to obtain the attribute information of the target object, the separate permission or separate consent of the target object will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the separate permission or separate consent of the target object, the necessary target object-related data used to enable the normal operation of the embodiment of the present disclosure will be obtained.
[0084] In the embodiments of the present disclosure, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories), or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.
[0085] To facilitate understanding of the technical solution provided by the embodiments of the present disclosure, some key terms used in the embodiments of the present disclosure are explained here:
[0086] Visual Entity Linking (VEL) refers to matching objects in an image with corresponding entities in a knowledge base.
[0087] In the related art, the image to be identified is usually encoded into global image features, and the query text describing the target object in the image to be identified is used as a visual cue, and then the entity identification corresponding to the image is predicted based on the global image features and the text features of the query text. However, global image features usually ignore local details in the image, resulting in information loss, thereby reducing the prediction accuracy of entity identification.
[0088] Based on this, the embodiments of the present disclosure provide an identification generation method, device, electronic device and storage medium, which can improve the prediction accuracy of the target entity identification.
[0089] Reference Figure 1 , Figure 1A schematic diagram of an optional implementation environment provided for an embodiment of the present disclosure, the implementation environment includes a terminal 101 and a server 102, wherein the terminal 101 and the server 102 are connected via a communication network.
[0090] Exemplarily, the server 102 can obtain a target image sent by the terminal, and then obtain local masks corresponding to each candidate object in the target image, and determine a query mask from multiple local masks, wherein the query mask is used to indicate a target object selected from multiple candidate objects; perform feature extraction based on the query mask and each local mask on the target image to obtain multiple mask visual features, and extract mask position features corresponding to the query mask and each local mask; splice each mask visual feature with the corresponding mask position feature to obtain multiple regional features; extract the first image feature of the target image, and splice the first image feature and multiple regional features to obtain a target splicing feature; perform text prediction based on the target splicing feature to generate a target entity identifier of the target object, and then the server 102 sends the target entity identifier to the terminal 101.
[0091] The server 102 obtains the local masks corresponding to each candidate object in the target image, and determines the query mask used to indicate the target object, and then obtains the query mask and the mask visual features corresponding to each local mask through feature extraction, and extracts the query mask and the mask position features corresponding to each local mask. The mask visual features can capture the pixel-level visual information of the local area where the corresponding object is located, and the mask position features can capture the pixel-level position information of the local area where the corresponding object is located. Then, each mask visual feature is spliced with the corresponding mask position features to obtain the regional features of each local area, which is equivalent to combining the pixel-level visual information of the local area and the corresponding pixel-level position information into pixel-level regional information, and then extracting the first image of the target image. The first image feature and multiple regional features are spliced into a target splicing feature, and then text prediction is performed based on the target splicing feature to generate a target entity identifier of the target object. In the text prediction process, by interacting with each feature in the target splicing feature, it is possible to pay attention to both the global visual information captured by the first image feature and the pixel-level regional information corresponding to each candidate object and the pixel-level regional information corresponding to the target object. Therefore, it is possible to effectively improve the understanding of the global image features and pixel-level details of the target image, thereby improving the prediction accuracy of the target entity identifier. In addition, using the pixel-level query mask as a visual cue can efficiently, flexibly and accurately refer to the target object, thereby further improving the prediction accuracy of the target entity identifier.
[0092] Server 102 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. In addition, server 102 may also be a node server in a blockchain network.
[0093] The terminal 101 may be a mobile phone, a computer, an intelligent voice interaction device, an intelligent home appliance, a vehicle-mounted terminal, etc., but is not limited thereto. The terminal 101 and the server 102 may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present disclosure.
[0094] Reference Figure 2 , Figure 2 An optional flow chart of an identification generation method provided for an embodiment of the present disclosure, the identification generation method can be executed by a server, or can also be executed by a terminal, or can also be executed by a server in conjunction with a terminal, the identification generation method includes but is not limited to the following steps 201 to 205.
[0095] Step 201: Obtain a local mask corresponding to each candidate object in a target image, and determine a query mask from multiple local masks.
[0096] The target image refers to an image that needs to be visually linked to entities. The target image may contain multiple candidate objects. For example, the candidate objects may include main objects such as animals, plants, and objects. The candidate objects may also include background objects such as natural landscapes, buildings, and city streets. Each local mask is used to indicate the corresponding candidate object. Each local mask can specify the area where the corresponding candidate object is located in the target image, which is equivalent to the local mask being able to refer to the corresponding candidate object; the query mask is used to indicate the target object selected from multiple candidate objects. The query mask can specify the local area where the target object is located in the target image, which is equivalent to the query mask being able to refer to the target object.
[0097] Specifically, each local mask can be a binary image of the same size as the target image. In any local mask, the focus area composed of all pixels with a pixel value of 1 is used to indicate the local area where the corresponding candidate object is located in the target image, and the non-focus area composed of all pixels with a pixel value of 0 is used to indicate the remaining area in the target image; optionally, the local mask can also be an image of other sizes, as long as the local mask can indicate the corresponding candidate object, and the embodiments of the present disclosure are not limited here.
[0098] Similarly, the query mask may be a binary image of the same size as the target image. In the query mask, the focus area composed of all pixels with a pixel value of 1 is used to indicate the local area where the target object is located in the target image, and the non-focus area composed of all pixels with a pixel value of 0 is used to indicate the rest of the target image. Optionally, the query mask may also be an image of other sizes, as long as the query mask can indicate the target object, which is not limited in the embodiments of the present disclosure.
[0099] Based on this, since the query mask is determined among multiple local masks, the query mask body is determined based on the selection of the relevant person, the query mask can characterize the reference intention of the relevant person, the target object is the object selected by the relevant person, and the target object can be any one of the multiple candidate objects.
[0100] Step 202: performing feature extraction based on the query mask and each local mask on the target image to obtain a plurality of mask visual features, and extracting mask position features corresponding to the query mask and each local mask.
[0101] Among them, feature extraction is performed on the target image based on the query mask and each local mask respectively, specifically referring to feature extraction based on the query mask and feature extraction based on each local mask respectively; assuming that the number of local masks is two, feature extraction is performed on the target image based on each local mask respectively, specifically referring to feature extraction based on the first local mask and feature extraction based on the second local mask; whenever feature extraction is performed on the target image, the corresponding mask visual features can be obtained.
[0102] Based on this, mask-based feature extraction is performed on the target image to obtain mask visual features, which is equivalent to extracting more abstract and more informative local visual features from the target image; since the query mask refers to the target object and the local mask refers to the corresponding candidate object, the mask visual features can capture the pixel-level visual information of the local area where the corresponding object is located; in addition, extracting the query mask and the mask position features corresponding to each local mask is equivalent to respectively determining the spatial position relationship between each mask, and since the query mask refers to the target object and the local mask refers to the corresponding candidate object, the mask position features can capture the pixel-level position information of the local area where the corresponding object is located.
[0103] In one possible implementation, the mask position features corresponding to the query mask and each local mask are extracted. Specifically, the query mask and each local mask are flattened and input into a position encoder for mapping to obtain the mask position features corresponding to the query mask and each local mask.
[0104] Step 203: Each mask visual feature is spliced with the corresponding mask position feature to obtain a plurality of regional features.
[0105] Among them, each mask visual feature is spliced with the corresponding mask position feature respectively, specifically, the mask visual feature corresponding to each local mask is spliced with the corresponding mask position feature, and the mask visual feature corresponding to the query mask is spliced with the corresponding mask position feature.
[0106] For example, for the query mask and any one of the local masks, the mask visual features corresponding to the mask can be spliced at the beginning or end of the corresponding mask position features to obtain the regional features corresponding to the mask; therefore, the query mask and any one of the local masks have corresponding regional features, and the regional features are the combination results of the mask visual features and the corresponding mask position features.
[0107] Based on this, the query mask and the regional features corresponding to each mask in each local mask can be obtained through splicing, that is, the regional features of the local area where each object is located are obtained, and the regional features of the local area are equivalent to the local features of the target image; specifically, splicing the mask visual features with the mask position features is equivalent to combining the pixel-level visual information and the corresponding pixel-level position information of the local area where the target object and each object in each candidate object are located, so that the pixel-level regional information of the local area where each object is located can be obtained. Therefore, each regional feature can characterize the regional details of the local area where the corresponding object is located, and subsequently the understanding of the pixel-level details of the target image can be improved based on the pixel-level regional information.
[0108] Step 204: extracting a first image feature of the target image, and splicing the first image feature and a plurality of regional features to obtain a target splicing feature.
[0109] Among them, the first image feature and multiple regional features are spliced. Specifically, the multiple regional features can be spliced in sequence to obtain a splicing result, and then the first image feature is spliced at the head end of the splicing result to obtain a target splicing feature. In addition, other splicing sequences can also be used to obtain the target splicing feature, which is not limited in the embodiments of the present disclosure.
[0110] Based on this, extracting the first image feature of the target image is equivalent to extracting a more abstract and more informative global image feature from the target image. The first image feature can capture the global visual information of the target image, and the first image feature and multiple regional features are spliced into a target splicing feature. Subsequently, based on the target splicing feature, both global visual information and pixel-level regional information can be focused on.
[0111] Step 205: Perform text prediction based on the target concatenation features to generate a target entity identifier of the target object.
[0112] Among them, the target entity identifier is used to indicate the target entity in the knowledge base. The target entity identifier is unique, that is, different entities in the knowledge base will correspond to different target entity identifiers. After the target entity identifier is generated, the target object can be linked to the target entity indicated by the target entity identifier based on the target entity identifier.
[0113] Specifically, the target entity identifier can be a single identifier or a sequence of multiple identifiers, which is not limited in the embodiments of the present disclosure. For example, the target entity identifier can be a sequence of [50, 10, 3], which includes 3 identifiers, namely 50, 10 and 3.
[0114] It is worth noting that the target entity identifier is used to indicate the entity in the knowledge base. In the knowledge base, each entity is usually designed to represent a unique object and corresponds to a globally unique label. Different labels can indicate different entities. For example, assuming that the name of an entity in the knowledge base is "golf course", the object corresponding to the entity is the golf course. The entity can be represented as e=Q1048XXX, where e is the entity and Q1048XXX is the label.
[0115] Based on this, in the text prediction process, by interacting with each feature in the target splicing feature, we can pay attention to the global visual information captured by the first image feature, the pixel-level area information corresponding to each candidate object, and the pixel-level area information corresponding to the target object. Therefore, we can effectively improve the understanding of the global image features and pixel-level details of the target image, thereby improving the prediction accuracy of the target entity identification. In addition, using the pixel-level query mask as a visual cue can efficiently, flexibly and accurately refer to the target object, thereby further improving the prediction accuracy of the target entity identification.
[0116] In one possible implementation, the first image feature and multiple regional features are spliced to obtain a target spliced feature. Specifically, the mask area of each local mask is determined separately, and the regional features corresponding to each local mask are spliced in order of size of the mask area to obtain a first spliced feature; the first image feature, the first splicing feature and the regional features corresponding to the query mask are spliced to obtain a target spliced feature.
[0117] Among them, the mask area of the local mask refers to the area of the focus area in the local mask. The focus area in the local mask is usually composed of pixels with all pixel values of 1. The focus area is used to indicate the local area where the corresponding candidate object is located in the target image; therefore, when the mask area is larger, the local area where the corresponding candidate object is located is larger, that is, the space occupied by the candidate object in the target image is larger, and the visual significance of the candidate object is higher. Conversely, when the mask area is smaller, the local area where the corresponding candidate object is located is smaller, that is, the space occupied by the candidate object in the target image is smaller, and the visual significance of the candidate object is lower.
[0118] Based on this, the regional features corresponding to each local mask are spliced in the order of the size of the mask area. Specifically, the local masks can be sorted in the order of the size of the mask area, which is equivalent to sorting the local masks in the order of visual significance, and then the regional features corresponding to the arranged local masks are spliced in turn to obtain the first splicing feature. Therefore, in the text prediction process, it is possible to focus on the pixel-level regional information of the local area where each candidate object is located in turn according to a fixed visual attention order, thereby effectively understanding the pixel-level details of the target image.
[0119] In addition, the first image feature, the first stitching feature and the regional features corresponding to the query mask are stitched together to obtain the target stitching feature. Therefore, in the text prediction process, it is possible to focus on the global visual information captured by the first image feature, thereby effectively improving the understanding of the global image features of the target image, and to focus on the pixel-level regional information of the local area where the target object is located, so as to achieve efficient, flexible and accurate reference to the target object through the query mask, thereby further improving the prediction accuracy of the target entity identification.
[0120] For example, each local mask is sorted in descending order according to the mask area. Then, in the first splicing feature, the space occupied by the candidate objects corresponding to the regional features spliced in sequence is getting smaller and smaller. In the text prediction process, the regional features corresponding to the candidate objects with larger occupied spaces are first focused on, and then the regional features corresponding to the candidate objects with smaller occupied spaces are focused on, which simulates the order of human visual attention. In the subsequent text prediction process, it ensures that the candidate objects with larger occupied spaces can receive more extensive attention, which usually helps to improve the prediction accuracy of the target entity identification.
[0121] For another example, each local mask is sorted in order of mask area from small to large. Then, in the first splicing feature, the space occupied by the candidate objects corresponding to the regional features spliced in sequence is getting larger and larger. In the text prediction process, we first focus on the regional features corresponding to the candidate objects that occupy a smaller space, and then focus on the regional features corresponding to the candidate objects that occupy a larger space. This helps to improve the prediction accuracy of the target entity identification in specific scenarios.
[0122] Specifically, since the first image feature can be regarded as a global feature of the target image, and the regional feature can be regarded as a local feature of the target image, the feature obtained by stitching the first image feature and the first stitching feature can be regarded as a comprehensive image feature. Taking the order of each local mask from large to small according to the mask area as an example, the formula for determining the comprehensive image feature is as follows:
[0123]
[0124] in, is the comprehensive image feature, is the first image feature, is the first local mask, is the second local mask, For the A local mask, is the regional feature corresponding to the first local mask, is the regional feature corresponding to the second local mask, For the The regional features corresponding to the local masks, is the first splicing feature formed by splicing multiple regional features. The first splicing feature can be regarded as a local feature sequence including multiple regional features, that is, , for to Any regional feature in is the length of the local feature sequence, that is is the number of local features in the local feature sequence. It can be seen that the first splicing feature can also be expressed as .
[0125] In addition, the size relationship between the mask areas of each local mask satisfies the following formula:
[0126]
[0127] in, For the The mask area of a local mask, For the The mask area of a local mask, that is, the mask area of the previous local mask is greater than or equal to the mask area of the local mask, which can ensure that the local masks are sorted in descending order according to the mask area, that is, It is obtained by sorting the features of each region in descending order based on the mask area.
[0128] It is worth noting that, assuming that the mask visual features and the mask position features are considered to be extracted by the mask-aware visual extractor, and each local mask is segmented by the semantic segmentation model, the formula for determining the first splicing feature is as follows:
[0129]
[0130] in, is the first splicing feature, is the target image, For any local mask, For local mask The corresponding mask visual features, For local mask The corresponding mask position features, is a mask-aware visual extractor, It is a semantic segmentation model.
[0131] It should be noted that, , as well as In is the target image, In Refers to the global (global), the used to indicate that the first image feature is a global feature of the target image, In Refers to local. It is used to indicate that the regional feature in the first stitching feature is a local feature of the target image.
[0132] In a possible implementation, the first image feature, the first stitching feature, and the regional feature corresponding to the query mask are stitched together to obtain a target stitching feature. Specifically, the first image feature, the first stitching feature, and the regional feature corresponding to the query mask are stitched together in sequence to obtain a target stitching feature. Based on this, the first stitching feature is spliced at the end of the first image feature. In the text prediction process, the global features in the target image used to capture overall information can be focused on first, and then the local features in the target image used to capture fine details can be focused on, which usually helps to improve the prediction accuracy of the target entity identification.
[0133] In a possible implementation, the target entity identifier is generated by the first large language model, and the first image feature, the first splicing feature, and the regional feature corresponding to the query mask are spliced to obtain the target splicing feature. Specifically, a prompt text is constructed to prompt the first large language model to generate the entity identifier; the text feature of the prompt text is extracted, and the first image feature, the text feature, the first splicing feature, and the regional feature corresponding to the query mask are spliced to obtain the target splicing feature.
[0134] Among them, the first language model belongs to the large language model (LLM), which is a deep learning model trained with a large amount of text data. It can generate natural language text or understand the meaning of language text. The large language model generally uses a recurrent neural network (RNN) or its variants, such as a long short-term memory network (LSTM) and a gated recurrent unit (GRU), to capture contextual information in a text sequence, thereby achieving tasks such as natural language text generation, language model evaluation, text classification, and sentiment analysis. In the field of natural language processing, large language models have been widely used, such as speech recognition, machine translation, automatic summarization, dialogue systems, intelligent question and answer, etc. Here, the target entity identifier belongs to the natural language text, and the first language model is used to process the task of generating the target entity identifier.
[0135] Based on this, the prompt text can be regarded as a prompt instruction (Prompt), which can be understood as a way to start the large language model. The prompt instruction can guide the large language model to generate content of a specific type, theme or format. Therefore, by constructing a prompt text that prompts the first large language model to generate an entity identifier, the target splicing feature includes not only the first image feature, the first splicing feature and the regional feature corresponding to the query mask, but also the text feature of the prompt text. The target splicing feature is subsequently used as the input of the first large language model. Under the guidance of the prompt text, the generation quality of the target entity identifier by the first large language model can be improved. In addition, by introducing the cross-attention interaction of pixel-level regional features in the first large language model, the first large language model can effectively improve its understanding of the global features and pixel-level details of the target image, thereby improving the prediction accuracy of the target entity identifier.
[0136] Specifically, the first image features, text features and regional features constitute the multimodal input of the large language model. Therefore, the first large language model can adopt a multimodal large language model (Multimodal Large Language Model, MLLM). The multimodal large language model has a better processing effect on multimodal input and can improve the prediction accuracy of the target entity identification. Among them, multimodality usually refers to information from different senses or sources, such as vision, hearing, touch, etc. The multimodal large language model is an extended form of the large language model. Compared with the large language model that can only process input data corresponding to the text modality, the multimodal large language model can not only process input data corresponding to the text modality, but also process input data corresponding to other modalities except the text modality. For example, other modalities include visual modalities, audio modalities or the results of a combination of multiple modalities, etc.
[0137] In one possible implementation, the prompt text may include instruction text and reference text, the instruction text is the prompt instruction, and the reference text is used to refer to the target object. Using the reference text as a text prompt can further accurately refer to the target object, thereby further improving the prediction accuracy of the target entity identification.
[0138] In a possible implementation, feature extraction is performed on the target image based on the query mask and each local mask to obtain a plurality of mask visual features. Specifically, multi-level feature extraction is performed on the target image to obtain multi-level visual features of the target image; mask pooling is performed on the multi-level visual features based on the query mask and each local mask to obtain a plurality of multi-level pooling features; and feature fusion is performed on each multi-level pooling feature to obtain a plurality of mask visual features.
[0139] Among them, feature extraction is performed on the target image based on the query mask and each local mask respectively, specifically referring to feature extraction based on the query mask and feature extraction based on each local mask respectively; assuming that the number of local masks is two, feature extraction is performed on the target image based on each local mask respectively, specifically referring to feature extraction based on the first local mask and feature extraction based on the second local mask; whenever feature extraction is performed on the target image, the corresponding mask visual features can be obtained.
[0140] Specifically, mask pooling refers to that in the target image, for any one of the query mask and each local mask, for the local area where the mask refers to the object, the multi-level visual features of all pixels in the local area are aggregated through the pooling operation to obtain the multi-level pooling features; mask pooling can be divided into mask operation and pooling operation. The mask operation is described in detail below.
[0141] First, for any one of the query mask and each local mask, the dimension of the mask is adjusted multiple times based on the number of levels of the multi-level visual features to obtain a target mask that matches the visual features of each level, wherein the mask is a binary image, and the focus area in the mask is composed of pixel points with a pixel value of 1, while other areas in the mask are composed of pixel points with a pixel value of 0. For example, assuming that the dimension of the visual features of a certain level is 128×128×16 and the dimension of the mask is 512×512, when adjusting the dimension, the dimension of the mask is first reduced to 128×128, and then the reduced mask is channel-copied to obtain a multi-channel image with 16 channels. The multi-channel image is defined as the target mask, and the dimension of the target mask is 128×128×16. Since the dimension of the target mask is the same as the dimension of the visual features of the level, the target mask matches the visual features of the level; therefore, when the number of levels of the multi-level visual features is five, the dimension of the mask can be adjusted five times to obtain target masks that match the visual features of the five levels.
[0142] Then, the visual features of this level are multiplied element-by-element with the target mask, and the visual features falling in the focus area of the target mask are retained, while the visual features falling in other areas of the target mask become 0, resulting in a focus feature with a dimension of 128×128×16.
[0143] The pooling operation is described in detail below. The pooling operation may adopt average pooling or maximum pooling, or other pooling methods, which are not limited in the embodiments of the present disclosure.
[0144] Taking the pooling operation using average pooling as an example, masking operations are performed on the visual features of each level to obtain the corresponding attention features. Assuming that the dimension of a certain attention feature is 128×128×16, the attention feature contains an initial feature map with a dimension of 128×128 of 16 channels. For each position in the initial feature map, the average value of the corresponding position in the 16 channels is taken as the feature value of the corresponding position in the new feature map, thereby generating a single-channel pooling feature map with a dimension of 128×128. After performing pooling operations on the attention features corresponding to the visual features of each level, the pooling feature maps corresponding to the visual features of each level can be obtained, and the pooling feature maps corresponding to the visual features of each level can form multi-level pooling features.
[0145] Among them, feature fusion is performed on each multi-level pooling feature respectively, which specifically refers to that for any multi-level pooling feature, all pooling feature maps in the multi-level pooling feature are subjected to feature fusion to obtain one of the mask visual features.
[0146] Based on this, in the multi-level feature extraction, the visual features of the target image at different levels can be extracted. High-level visual features are usually more abstract and more informative than low-level visual features. Multi-level visual features provide richer and more diverse data representations, which can learn and express the target image more comprehensively and effectively, and obtain appropriate multi-level visual features. Then, in the mask pooling process, the pixel-level visual information of the specific area in the target image can be accurately captured, and the visual features can be accurately extracted from the local area where the target object and each candidate object are located, and the appropriate multi-level pooling features can be obtained. Then, in the feature fusion process, the pooling feature maps of different levels can be integrated to obtain the appropriate mask visual features, and fine-grained visual understanding can be achieved. The integrity of the pixel-level visual information in the specific area can be enhanced. In the subsequent text prediction process, the understanding of the pixel-level details of the target image can be effectively improved, thereby improving the prediction accuracy of the target entity identification.
[0147] In a possible implementation, extracting the first image feature of the target image may specifically involve performing multi-level feature extraction on the target image to obtain multi-level visual features of the target image; and inputting the multi-level visual features into a first multi-layer perceptron for mapping to obtain the first image feature.
[0148] Based on this, the first multi-layer perceptron is used to perform multi-layer perception processing, and the multi-level visual features are further abstracted through the multi-layer perception processing, so that a higher-level data representation can be learned, thereby improving the prediction accuracy of the target entity identification; the first multi-layer perceptron and the first large language model can be jointly trained. In addition, when extracting the mask visual features and the first image features, the same visual encoder can be used for multi-level feature extraction. By sharing the feature mapping of the visual encoder, additional calculation and parameter overhead can be reduced.
[0149] In one possible implementation, the reference photo Figure 3 , Figure 3 An optional flow chart for determining mask visual features provided in an embodiment of the present disclosure, wherein feature fusion is performed on each multi-level pooling feature to obtain multiple mask visual features. Specifically, for any multi-level pooling feature, sub-features of each level in the multi-level pooling feature are mapped to obtain multiple intermediate features of the same dimension, and each intermediate feature is feature fused to obtain a fused feature; multi-layer perception processing is performed on each fused feature to obtain multiple mask visual features.
[0150] Among them, the various intermediate features are subjected to feature fusion. Specifically, the various intermediate features may be summed up, or the various intermediate features may be spliced, which is not limited in the embodiments of the present disclosure.
[0151] Based on this, the sub-features of each level in the multi-level pooling features are the above-mentioned pooling feature maps. Since the dimensions of sub-features at different levels are usually different, before feature fusion, it is necessary to map each sub-feature to an intermediate feature of the same dimension, and then perform feature fusion on the intermediate features of the same dimension to obtain fused features, which can effectively integrate sub-features of different levels. The fused features are further abstracted through multi-layer perception processing, and more advanced data representations can be learned. Finally, suitable mask visual features are generated to provide richer and more accurate visual information for text prediction, which helps to improve the understanding of pixel-level details of the target image, thereby improving the prediction accuracy of the target entity identification.
[0152] Specifically, when the target entity identification is generated by the first largest language model, the multi-layer perception processing can map the fused features with visual information into masked visual features with language information, which is equivalent to mapping the fused features to a feature space that matches the text embedding space, thereby realizing the conversion of features between different representation spaces, and can effectively integrate the visual information and text information input into the first largest language model, which helps to improve the first largest language model's understanding of the pixel-level details of the target image, thereby improving the prediction accuracy of the target entity identification.
[0153] In addition, each intermediate feature can be mapped by the corresponding first linear layer; and each fused feature is subjected to multi-layer perception processing respectively. Specifically, each fused feature can be input into the second multi-layer perceptron for mapping, so as to obtain the mask visual features corresponding to each fused feature. The first linear layer, the second multi-layer perceptron and the first large language model can be trained jointly.
[0154] In a possible implementation, a local mask corresponding to each candidate object in a target image is obtained, and a query mask is determined from multiple local masks. Specifically, a target image and a prompt mark of the target image are obtained; the target image is segmented to obtain a local mask corresponding to each candidate object, and a query mask is determined from each local mask based on the prompt mark.
[0155] Among them, the target image includes multiple candidate objects, and the prompt mark is used to indicate the target object selected from the multiple candidate objects. The prompt mark refers to a mark drawn on the target image. For example, the prompt mark can be a point, line, box, smear area, etc. drawn on the target image.
[0156] Specifically, the target image is segmented. Specifically, the category distribution probability of each pixel in the target image can be predicted first. The category distribution probability includes the matching probability value of the pixel belonging to each candidate object. For any pixel, the candidate object with the highest matching probability value is determined as the candidate object matched by the pixel. The local area where each candidate object is located is composed of all the matching pixels, respectively. Then, based on the candidate object matched by each pixel, the local area where each candidate object is located can be accurately segmented in the target image. Then, based on the local area where each candidate object is located, the local mask corresponding to each candidate object can be accurately determined.
[0157] For example, for any candidate object, an initial image identical to the target image is created. In the initial image, the pixel values of the pixels in the local area where the candidate object is located are assigned to 1, and the pixel values of the pixels in other areas of the initial image are assigned to 0, to obtain the local mask corresponding to the candidate object. The local mask corresponding to the candidate object can also be determined by other methods, which are not limited to the embodiments of the present disclosure.
[0158] Based on this, by segmenting the target image, the local mask corresponding to each candidate object can be accurately obtained. The prompt mark can be drawn by the relevant personnel, and the prompt mark can represent the reference intention of the relevant personnel. Therefore, the target object indicated by the prompt mark is specifically the object selected by the relevant personnel. The query mask is determined in the local mask based on the prompt mark, so that the query mask can also represent the reference intention of the relevant personnel. By using the pixel-level query mask as a visual cue, the reference target object can be efficiently, flexibly and accurately represented, thereby further improving the prediction accuracy of the target entity identification.
[0159] Specifically, refer to Figure 4 , Figure 4 An optional process diagram for generating a mask provided in an embodiment of the present disclosure may input a target image and a prompt mark into a target mask generation model for mask prediction to generate a local mask and a query mask. For example, the target mask generation model may adopt a Segment Anything Model (SAM), a Fast SegmentAnything Model (FastSAM) and other models, which are not limited in the embodiment of the present disclosure.
[0160] In a possible implementation, the query mask is determined in each local mask based on the hint mark. Specifically, when the hint mark is a mark point, the query mask is determined in each local mask based on a positional relationship between the mark point and each local mask.
[0161] Alternatively, when the hint is marked as a marked box, the query mask is determined in each local mask based on a degree of matching between the marked box and a mask boundary of each local mask;
[0162] Alternatively, when the hint is marked as a labeled region, the query mask is determined in each local mask based on the matching degree between the labeled region and each local mask.
[0163] Among them, the number of marking points can be one or more. When the number of marking points is one, the marking point is located within the mask boundary of the query mask; when the number of marking points is multiple, the local mask containing the most marking points in the focus area can be determined as the query mask, and the query mask can also be determined by other methods, which is not limited in the embodiments of the present disclosure.
[0164] Based on this, when processing the marking point, since the marking point is a pixel point in the area where the target object is located on the target image, by determining the positional relationship between the marking point and each local mask, it is possible to determine whether the marking point is located within the mask boundary of the local mask, that is, to determine whether the focus area in the local mask contains the marking point, and then the local mask containing the marking point in the focus area can be determined as the query mask; when processing the marking box, since the selection area of the marking box can usually cover the area where the target object is located, the matching degree between the marking box and the mask boundary of each local mask can be calculated, and then the local mask with the highest matching degree can be determined as the query mask; similarly, when processing the marking area, the marking area is equivalent to graffiti, and since the marking area can usually cover the area where the target object is located, the matching degree can also be calculated, and the local mask with the highest matching degree can be determined as the query mask; therefore, the above three processing methods all enable the query mask to effectively cover the area where the target object is located, thereby ensuring that the query mask can efficiently, flexibly and accurately represent the target object, thereby further improving the prediction accuracy of the target entity identification.
[0165] It should be noted that the prompt mark can be determined in a variety of ways. For example, the target image and the image marking control are displayed in the display interface, and the prompt mark is determined in response to the interaction of the image marking control. The image marking control may specifically include an image click control, an image frame selection control, an image smear control, and the like. Exemplarily, when the image marking control is an image click control, in response to the interaction of the image click control, it can detect the pixel point clicked by the relevant person on the target image, and then determine the pixel point as the marking point; when the image marking control is an image frame selection control, in response to the interaction of the image frame selection control, it can detect the bounding box drawn by the relevant person on the target image, and then determine the bounding box as the marking box; when the image marking control is an image smear control, in response to the interaction of the image smear control, it can detect the smear area drawn by the relevant person on the target image, and then determine the smear area as the marking area.
[0166] For another example, a text input box is displayed in the display interface. In response to the interaction of the text input box, the content input by the relevant personnel in the text input box can be obtained, and then the prompt mark is determined based on the content input in the text input box. Exemplarily, assuming that the target image is represented as a two-dimensional matrix of x rows and y columns, the content input in the text input box can be "the pixel point located at the x'th row and y'th column in the target image", and then based on the content input in the text input box, the pixel point located at the x'th row and y'th column in the target image is determined as the mark point, where x'≤x, y'≤y.
[0167] In a possible implementation, based on the degree of matching between the marked area and each local mask, a query mask is determined in each local mask. Specifically, the intersection-and-union ratio between the marked area and each local mask is determined, that is, the matching degree is specifically the intersection-and-union ratio, and then the local mask with the largest intersection-and-union ratio is determined as the query mask; or, the center distance between the marked area and each local mask is determined, that is, the matching degree is specifically the center distance, and then the local mask with the largest center distance is determined as the query mask; other methods may also be used to determine the query mask, which is not limited in the embodiments of the present disclosure.
[0168] Specifically, similar to the processing method of the marked area, the matching degree between the marked box and the mask boundary of each local mask can also be determined by calculating the intersection-over-union ratio or the center distance.
[0169] In a possible implementation, the target entity identifier is generated by a first large language model, and the first large language model is trained through the following steps: obtaining a sample image and a second mask corresponding to a sample object in the sample image, segmenting the sample image, and obtaining a first mask corresponding to each visual object in the sample image, wherein the sample object is one of multiple visual objects; extracting a second image feature of the sample image, performing feature extraction based on the second mask and each first mask on the sample image to obtain multiple sample visual features, and extracting sample position features corresponding to the second mask and each first mask; splicing the second image feature, the sample visual feature, and the sample position feature and inputting them into the first large language model for text prediction to generate a predicted probability distribution, wherein the predicted probability distribution is used to determine the entity identifier of the sample object; obtaining a first entity identifier of a sample entity linked to the sample image, determining a model loss based on the predicted probability distribution and the first entity identifier, and training the first large language model based on the model loss.
[0170] It is worth noting that, similar to the target image, the sample image refers to the image that needs to be visually linked; similar to the local mask, each first mask is used to indicate the corresponding visual object, and the first mask can specify the local area where the corresponding visual object is located in the sample image; similar to the query mask, the second mask is used to indicate the sample object, and the second mask can specify the local area where the sample object is located in the sample image; similar to the target object, the sample object is equivalent to an object selected from multiple visual objects; similar to the mask visual feature, the sample visual feature can capture the pixel-level visual information of the local area where the corresponding object is located; similar to the mask position feature, the sample position feature can capture the pixel-level position information of the local area where the corresponding object is located.
[0171] Among them, the sample entity is used to indicate the sample object. The sample entity can be the label of the entity in the knowledge base. In the knowledge base, each entity is usually designed to represent a unique object and corresponds to a globally unique label. Different labels can indicate different entities. For example, the sample entity can be e=Q10000XX, where e is the entity and Q10000XX is the label.
[0172] Among them, the first entity identifier may include one sample identifier or multiple sample identifiers; when the first entity identifier is a sequence composed of multiple sample identifiers, the large language model will generate a prediction probability distribution corresponding to each sample identifier, each prediction probability distribution includes a prediction probability value of each candidate identifier, and the entity identifier of the sample object can be determined based on the candidate identifier with the highest prediction probability value; when the first entity identifier is a single sample identifier, the prediction probability distribution includes the prediction probability value of each candidate identifier.
[0173] Specifically, the first entity identifier is used to indicate the sample entity of the sample object. The sample identifier may be a word segment or an index of a word segment. The sample identifier may also be in other forms, as long as the first entity identifier can indicate the sample entity of the sample object. The embodiments of the present disclosure are not limited here.
[0174] For example, when the sample identifier is a segmentation word, the candidate identifier is also a segmentation word, and all segmentations are recorded in the vocabulary. The predicted probability distribution includes the predicted probability value of each segmentation word in the vocabulary. For example, the first entity identifier is [_course][olf][_G], and the first entity identifier includes 3 sample identifiers, the first sample identifier is [_course], the second sample identifier is [olf], and the sample identifier is [_G].
[0175] For another example, when the sample identifier is the index of a word segment, the candidate identifier is also the index of the word segment, all word segments and the corresponding indexes can be recorded in the vocabulary, and the predicted probability distribution includes the predicted probability values of the indexes of each word segment in the vocabulary. Assuming that the vocabulary includes 100 word segments and corresponding indexes, the predicted probability distribution includes 100 predicted probability values; wherein the index is used to indicate the position of the corresponding word segment in the word embedding matrix, and the word embedding matrix includes the word embedding vectors of each word segment in the word embedding space. Assuming that the first entity identifier is [10,40,5], the first entity identifier includes 3 sample identifiers, namely 10, 40 and 5, respectively. Sample identifier 10 is used to indicate the 10th position in the word embedding matrix, sample identifier 40 is used to indicate the 40th position in the word embedding matrix, and sample identifier 5 is used to indicate the 5th position in the word embedding matrix. At this time, the first entity identifier can exist in the form of an integer sequence, which can also be called an integer code, thereby realizing the representation of each entity in the knowledge base by a compact integer code.
[0176] Based on this, by obtaining a sample image, and then obtaining a first mask corresponding to each visual object in the sample image by segmenting the sample image, and also obtaining a second mask corresponding to the sample object in the sample image, and then obtaining the second mask and the sample visual features corresponding to each first mask by feature extraction, and extracting the second mask and the sample position features corresponding to each first mask, and performing text prediction through the first large language model, it is possible to generate a predicted probability distribution for determining an entity identifier, and then determine the sample probability distribution based on the first entity identifier of the sample entity, use the sample probability distribution as label data, and then determine the model loss based on the difference between the predicted probability distribution and the sample probability distribution, and perform supervised learning on the first large language model based on the model loss, and reduce the model loss through iterative training, so that the first large language model can improve the understanding of the global features and pixel-level details of the target image, thereby improving the prediction accuracy of the target entity identifier.
[0177] In a possible implementation, the sample image, the second mask and the first entity identifier are all obtained from a data set. Before obtaining the sample image and the second mask corresponding to the sample object in the sample image, the identifier generation method also includes: obtaining multiple original images and query texts corresponding to each original image, and determining the identification information corresponding to each original image according to each original image and the corresponding query text; obtaining multiple candidate entities, and determining the linked entity corresponding to each original image in the multiple candidate entities based on each identification information; determining the annotation mask of each original image based on each original image and the corresponding query text; and associating and storing each original image, the corresponding annotation mask and the corresponding linked entity in the data set.
[0178] Among them, the original image refers to the image that needs to be visually linked. The original image usually includes multiple objects, but only the focus object in the original image needs to be visually linked. Similar to the target object, the focus object is equivalent to an object selected from multiple objects in the original image. The query text is used to prompt the identification of the corresponding focus object in the original image. The query text can represent the intention of referring to the focus object. The annotation mask is used to indicate the corresponding focus object. The annotation mask can specify the local area where the focus object is located in the original image.
[0179] Specifically, each candidate entity can be a label of an entity in the knowledge base. In the knowledge base, each entity is usually designed to represent a unique object and corresponds to a globally unique label. For example, a candidate entity can be e=Q10000XX, and another candidate entity can be e=Q20000XX, where e is the entity, and Q10000XX and Q20000XX are both labels; therefore, the linked entity can also be a label of an entity in the knowledge base.
[0180] The sample image is sampled from a plurality of original images, the second mask is a labeling mask corresponding to the sample image, and the sample entity is a link entity linked to the sample image.
[0181] Based on this, according to the additional semantic context provided by the original image and the corresponding query text, accurate recognition information can be determined, and then based on the recognition information, accurate linked entities can be determined. There is a correlation between the original image, the corresponding annotation mask and the corresponding linked entity. Therefore, the original image, the corresponding annotation mask and the corresponding linked entity can be used as a group of training samples, and the original image, the corresponding annotation mask and the corresponding linked entity are associated and stored in the data set to achieve linking the original image with the corresponding linked entity, so that the object of interest indicated by the annotation mask has a link relationship with the corresponding linked entity; since the data set usually includes multiple groups of training samples, training samples can be sampled from the data set during the training process, and then the original image in the sampled training sample is determined as the sample image, the annotation mask in the sampled training sample is determined as the second mask, and the linked entity in the sampled training sample is determined as the sample entity, ensuring that there is a correlation between the sampled sample image, the second mask and the sample entity, thereby ensuring the effective progress of the training process.
[0182] Specifically, the method of sampling training samples from the data set can be random sampling, uniform sampling, etc., which is not limited in the embodiments of the present disclosure; during the training process, the first language model can be pre-trained using the training samples collected from the data set, so that the first language model learns the knowledge of entities and entity identifiers, which helps to generate effective target entity identifiers during the reasoning process. Then, the first language model is fine-tuned using the training samples of the downstream scene to improve the ability of the first language model to perform fine-grained visual entity linking.
[0183] Specifically, the link entity may be determined in a variety of ways, and the first way of determining the link entity is described in detail below.
[0184] In one possible implementation, the recognition information is a recognition coding result. According to each original image and the corresponding query text, the recognition information corresponding to each original image is determined respectively. Specifically, each original image and the corresponding query text can be input into a multimodal coding model for encoding to obtain a recognition coding result corresponding to each original image. The multimodal coding model can be a contrastive language-image pre-training model (Contrastive Language-Image Pre-training, CLIP), a path language image model (Pathways Language and Image model, PaLI), etc., and the embodiments of the present disclosure are not limited thereto.
[0185] Then, based on each piece of recognition information, the linked entity corresponding to each original image is determined from multiple candidate entities. Specifically, the candidate name text of each candidate entity in the knowledge base and the corresponding candidate image are input into the multimodal coding model for encoding to obtain the candidate coding result corresponding to each candidate entity. For any original image, the similarity between the recognition coding result corresponding to the original image and each candidate coding result is determined respectively, and the candidate coding result with the highest similarity is determined as the target coding result, and the candidate entity corresponding to the target coding result is determined as the linked entity corresponding to the original image. Based on this, the linked entity corresponding to the original image can be quickly and accurately determined.
[0186] It should be noted that in addition to storing the labels of candidate entities, the knowledge base also stores the relevant text of the candidate entities and the corresponding candidate images. The candidate images are used to display the candidate entities. The relevant text of the candidate entities includes the name text and description text of the candidate entities. The name text is used to indicate the entity and is readable, and the description text is used to describe the candidate entity.
[0187] The second method for determining the link entity is described in detail below.
[0188] In another possible implementation, the identification information is an identification name text. According to each original image and the corresponding query text, the identification information corresponding to each original image is determined respectively. Specifically, each original image and the corresponding query text can be input into a multimodal coding model for encoding to obtain an identification coding result corresponding to each original image; the identification coding result is decoded to obtain the identification name text corresponding to the original image.
[0189] Then, based on each piece of recognition information, the linked entity corresponding to each original image is determined among multiple candidate entities. Specifically, for any original image, based on the recognition name text corresponding to the original image, a consistent target name text is retrieved from the candidate name texts of each candidate entity in the knowledge base, and the candidate entity corresponding to the target name text is determined as the linked entity corresponding to the original image. Based on this, the linked entity corresponding to the original image can be accurately determined.
[0190] It should be noted that the link entity may also be determined in other ways, which are not limited in the embodiments of the present disclosure.
[0191] In a possible implementation, based on each original image and the corresponding query text, the annotation mask of each original image is determined respectively. Specifically, each query text can be input into the second largest language model for text prediction to generate a summary text corresponding to each original image; object detection is performed based on each original image and the corresponding summary text to generate an original bounding box corresponding to each original image, wherein the original bounding box is used to indicate the corresponding object of interest; each original image and the corresponding original bounding box is input into the first mask generation model for mask prediction to generate an annotation mask corresponding to each original image.
[0192] Among them, the second largest language model belongs to the large language model. The second largest language model is used to process the task of extracting referential expressions from text. The second largest language model can generate corresponding summary text based on the query text. The summary text can describe the location of the object of interest or the relationship with other objects. For example, assuming the query text is "What is the brown object placed on the chair", the corresponding summary text may be "the brown object on the chair".
[0193] Based on this, object detection is performed based on the original image and the corresponding summarized text, and an accurate original bounding box can be generated so that the original bounding box can cover the area where the object of interest is located. The original image and the corresponding original bounding box are then input into the first mask generation model for mask prediction. The original bounding box provides accurate position hints, which can improve the quality of the annotation mask generated by the first mask generation model, thereby effectively improving the annotation success rate of the annotation mask.
[0194] Specifically, refer to Figure 5 , Figure 5 An optional flowchart of updating a data set provided in an embodiment of the present disclosure.
[0195] First, identification information is determined according to the original image and the corresponding query text, and the corresponding link entity is determined based on the identification information;
[0196] Then, the query text is input into the second largest language model for text prediction to generate a summarized text, and the original image and the corresponding summarized text are input into the object prediction model for object detection to generate an accurate original bounding box; wherein, the object detection model may adopt the Grounding DINO model or other models, which is not limited in the embodiments of the present disclosure.
[0197] Then, the original image and the corresponding original bounding box are input into the first mask generation model for mask prediction to generate a labeled mask, wherein the first mask generation model may adopt a Segment Anything Model (SAM), a FastSegment Anything Model (FastSAM) and other models, which are not limited in the embodiments of the present disclosure.
[0198] Then, the original image, the corresponding annotation mask, and the corresponding linked entity are associated and stored in the dataset.
[0199] In one possible implementation, referring to Figure 6 , Figure 6 An optional architectural schematic diagram for removing original images provided in an embodiment of the present disclosure, wherein each original image and the corresponding original bounding box are respectively input into a first mask generation model for mask prediction, and after generating an annotation mask corresponding to each original image, the identification generation method further includes: obtaining a reference name text corresponding to each linked entity, wherein the reference name text is used to indicate the name of the reference entity, and the level of the reference entity in the knowledge base is higher than the level of the linked entity in the knowledge base; each original image and the corresponding reference name text are respectively input into a second mask generation model for mask prediction, and a reference mask corresponding to each original image is generated; determining the degree of matching between each annotation mask and the corresponding reference mask, and obtaining a target matching degree corresponding to each annotation mask; when the target matching degree is less than a preset matching degree threshold, removing the original image corresponding to the target matching degree.
[0200] Among them, the type of the knowledge base may be a knowledge graph, and different entities in the knowledge base may have a hierarchical relationship. Suppose the knowledge base includes an entity e=Q73XX named "mammal", and the knowledge base also includes an entity e=Q3009XX named "cat". Since "cat" belongs to "mammal", it can be set in the knowledge base that entity e=Q73XX and entity e=Q3009XX have a hierarchical relationship, and the level of entity e=Q73XX is higher than that of entity e=Q3009XX; therefore, assuming that the linked entity is entity e=Q3009XX, since the level of entity e=Q73XX is higher than that of entity e=Q3009XX, entity e=Q73XX can be determined as a reference entity, and "mammal" can be determined as a reference name text, and a higher-level entity can also be determined as a reference entity, and the embodiments of the present disclosure are not limited here.
[0201] Among them, the second mask generation model may adopt the Segment Everything Everywhere All at Once Model (SEEM) model, or other models, which is not limited in the embodiments of the present disclosure.
[0202] It should be noted that the matching degree threshold can be obtained through the prediction of the trained first regression model or determined through multiple experiments, and the embodiments of the present disclosure are not limited thereto.
[0203] Based on this, the original image and the corresponding reference name text are input into the second mask generation model for mask prediction. By expanding the semantic scope of the object of interest through the reference name text, the quality of the reference mask generated by the second mask generation model can be improved. Then, the degree of matching between the annotation mask and the corresponding reference mask is determined. When the target matching degree is less than the preset matching degree threshold, it means that the first mask generation model may have generated a low-quality annotation mask. By eliminating the original image corresponding to the target matching degree, it is equivalent to filtering the original image, which can avoid storing low-quality training samples in the data set, thereby effectively improving the training quality of the first language model. By using the reference mask generated by the second mask generation model as a supplementary strategy, inappropriate annotation masks are filtered to deal with the error propagation problem that may occur in the generation process of the first mask generation model.
[0204] Specifically, similar to the processing method of the marked area, the matching degree between each annotation mask and the corresponding reference mask can also be determined by calculating the intersection-over-union ratio or the center distance.
[0205] In one possible implementation, in addition to inputting the reference name text into the second mask generation model, the name text, description text, category text, etc. of the linked entity may also be input into the second mask generation model, which can further improve the quality of the reference mask generated by the second mask generation model.
[0206] In one possible implementation, each original image and the corresponding query text can be divided into different original subsets according to different task objectives, data processing requirements or application scenarios. Assume that the original subset includes an entity subset and a query subset. The task goal of the entity subset is to enable the model to recognize specific entities from the original image. The query text of the entity subset is usually used to describe the type of entity that should be recognized from the image. For example, the original image shows a cat, and the query text can be "identify the animal in the image"; and the task goal of the query subset is to enable the model to not only recognize the entities in the image, but also understand the context and intent of the query text. For example, the original image shows a girl wearing a headband, and the query text can be "what is the little girl wearing in her hair".
[0207] Specifically, the original subset may also include a human-annotated subset. The task objective of the human-annotated subset may be the same as the task objective of the entity subset or the query subset. The original images in the human-annotated subset are manually screened, and the query text in the human-annotated subset is manually set, which can ensure the quality of the human-annotated subset.
[0208] In one possible implementation, after determining the degree of matching between each annotation mask and the corresponding reference mask and obtaining the target matching degree corresponding to each annotation mask, the identification generation method also includes: when the target matching degree is greater than or equal to the matching degree threshold, and the original image and the corresponding query text are from the entity subset, there is no need to adjust the annotation mask; or, when the target matching degree is greater than or equal to the matching degree threshold, and the original image and the corresponding query text are from the query subset, each annotation mask is replaced with the corresponding reference mask.
[0209] Based on this, the accuracy of the annotation mask of each original image in the entity subset can be improved, and the accuracy of the annotation mask of each original image in the query subset can also be improved.
[0210] In a possible implementation, before each original image, the corresponding annotation mask, and the corresponding link entity are associated and stored in the data set, the identification generation method also includes: counting the number of connected areas in each annotation mask to obtain the number of areas corresponding to each annotation mask; when the number of areas is greater than a preset number threshold, discarding the original image corresponding to the number of areas.
[0211] Among them, the connected area is an area composed of pixels with the same pixel value and adjacent to each other in the mark mask. The annotation mask can be a binary image. The pixel value of the pixel points in the connected area is 1, and the pixel value of the pixel points in the area outside the connected area is 0. The connected area is equivalent to the focus area in the mark mask.
[0212] Usually, most or all of the area occupied by an object in an image is connected. Therefore, when the number of connected areas is too large, it means that the annotation mask may indicate multiple objects in the original image, that is, the annotation mask cannot accurately indicate only the object of interest in the original image.
[0213] Based on this, when the number of regions is greater than the quantity threshold, it can be considered that the number of connected regions is too large, and the annotation mask is defined as a low-quality mask. By eliminating the corresponding original image, it is possible to avoid storing low-quality training samples in the data set, thereby effectively improving the training quality of the first language model.
[0214] It should be noted that the quantity threshold can be obtained through the prediction of the second regression model after training or determined through multiple experiments, and the embodiments of the present disclosure are not limited thereto.
[0215] In one possible implementation, eliminating original images is equivalent to filtering training samples. Since the objects of interest that do not belong to visual entities cannot be effectively indicated by the annotation mask, in addition to filtering training samples based on the number of regions, filters can be used to filter training samples. For example, by filtering the query text through a text filter, original images whose objects of interest do not belong to visual entities can be eliminated. For example, if the object of interest of the original image is a certain meeting, the original image is eliminated.
[0216] Specifically, refer to Figure 7 , Figure 7 An optional distribution diagram of entity categories in multiple sample sets provided in an embodiment of the present disclosure.
[0217] Among them, the sample set composed of all training samples before filtering and without annotation masks is defined as the initial sample set, and the sample set composed of all training samples after filtering is defined as the optimized sample set. It can be seen that in each entity category, the number of samples in the optimized sample set is smaller than the number of samples in the initial sample set, especially in the categories of place, building and sports. The objects of interest in the original image usually do not belong to visual entities. Filtering is performed using filters, so that the number of samples in the optimized sample set is much smaller than the number of samples in the initial sample set.
[0218] refer to Figure 8 , Figure 8 An optional pie-shaped diagram of entity categories in the optimized sample set provided in an embodiment of the present disclosure.
[0219] It can be seen that in the optimized sample set, the proportion of other categories, animal categories and plant categories is relatively large. The objects of interest in these categories usually belong to visual entities, and the objects of interest belonging to visual entities can be effectively indicated by the annotation mask. Therefore, by optimizing the training samples in the sample set, the first language model can be effectively trained.
[0220] In addition, refer to Fig. 9 , Fig. 9 An optional distribution diagram of the area ratio of the annotation mask provided in the embodiment of the present disclosure.
[0221] Among them, the area ratio of the annotation mask is specifically the ratio between the area of the annotation mask and the area of the corresponding original image. It can be seen that the distribution curve of the area ratio is smooth as a whole, and when the area ratio exceeds 95%, the frequency will increase slightly. This is because there are dense groups of objects in some original images. For example, the original image includes groups of vegetation. During the annotation process, it will be regarded as a coherent object, and the annotation mask will be used to indicate the groups of vegetation.
[0222] An optional data statistics of the initial sample set and the optimized sample set is described in detail below.
[0223] For the initial sample set, the data statistics are shown in Table 1 below:
[0224] Table 1
[0225]
[0226] As can be seen from Table 1, the initial sample set may include a training set, a validation set, a test set, and a manually annotated set. The initial sample set contains 5,214,965 images with 5,245,421 annotations, covering a total of 20,077 entities. The manually annotated set refers to a manually annotated data set, “visible” means that the object of interest in the image usually belongs to a visual entity, and “invisible” means that the object of interest in the image usually does not belong to a visual entity.
[0227] For the optimized sample set, the data statistics are shown in Table 2 below:
[0228] Table 2
[0229]
[0230] As can be seen from Table 2, the optimized sample set contains 1,965,145 training samples with labeled masks, which can effectively train the first language model.
[0231] In a possible implementation, the first entity identifier of the sample entity linked to the sample image is obtained, specifically, the sample name text of the sample entity linked to the sample image is obtained, and the sample name text is segmented to obtain multiple first participles; the frequency of occurrence of each first participle in the knowledge base is determined, and the first participles are sorted in order from small to large based on the order of their occurrence frequencies, and the first participles arranged in the first L positions are determined as second participles; based on each second participle, the first entity identifier of the sample entity is determined.
[0232] Among them, the sample name text is used to indicate the sample entity and is readable. The sample name text can be segmented in a variety of ways, such as inputting the sample name text into a text segmenter for segmentation, or segmenting the sample name text according to a fixed length, which is not limited in the embodiments of the present disclosure.
[0233] Wherein, the knowledge base includes multiple candidate entities and corresponding candidate name texts, and the candidate name texts are text corpora in the knowledge base. When determining the frequency of occurrence of the first participle in the knowledge base, all text corpora in the knowledge base can be firstly segmented, and then a segmentation union is constructed through all segmentation structures, and then the frequency of occurrence of the first participle in the segmentation union is used as the frequency of occurrence of the first participle in the knowledge base. Optionally, in addition to the candidate name texts, the text corpora may also include description texts of the candidate entities, or other texts, which are not limited in the embodiments of the present disclosure.
[0234] Wherein, L is a positive integer, and each first participle is sorted in order of appearance frequency from small to large, and the first participle arranged in the first L positions is determined as the second participle, which is equivalent to taking the first L participles as the second participle after sorting each first participle in ascending order of appearance frequency in the participle union; usually, L is less than or equal to the number of first participles. When the number of L is greater than the number of first participles, all sorted first participles are determined as the second participle. The value of L is usually small. For example, L can be 4, which can improve the decoding efficiency of the first language model.
[0235] Based on this, the sample name text is segmented to obtain the second segmentation, which can decompose the long text sample name text into smaller units, so that the first language model can learn and understand the structure and meaning of the language more effectively, and can also improve the processing efficiency. By taking the first L segmentations after sorting the first segmentations in ascending order of the frequency of occurrence in the segmentation union, it is equivalent to selecting the segmentation with the lowest frequency of occurrence, so that the first entity identifier determined by all the second segmentations is more unique, which can reduce confusion, and the second segmentation with a lower frequency of occurrence is ranked in the front, so that the first language model can decode a more distinctive result first, thereby improving the prediction accuracy of the entity identifier; in addition, since the first entity identifier is determined by at most L second segmentations, the length of the entity identifier is limited, which can improve the decoding efficiency of the first language model.
[0236] Specifically, the first entity identifier may include one sample identifier or multiple sample identifiers; the sample identifier may be a word segment or an index of a word segment, and the sample identifier may also be in other forms, which is not limited in the embodiments of the present disclosure.
[0237] Taking the candidate identifier as a word segment as an example, when the first entity identifier is a sequence composed of multiple sample identifiers, the formula for determining the first entity identifier is as follows:
[0238]
[0239] in, is the first entity identifier, is a text tokenizer, is the sample name text, For multiple first participles, For the knowledge base, The first candidate entities, It refers to The candidate name text of the candidate entity is segmented. It refers to the word segmentation union of the word segmentation results of the candidate name texts of all candidate entities in the knowledge base. Used to take the first L words in ascending order of word frequency, It means that each first participle is in the participle union Take the first L words in ascending order of frequency of occurrence, that is Including each second participle, each second participle can be used as a sample identifier, so the first entity identifier Consists of all sample identifiers.
[0240] For example, assuming that the sample name text is Golf course, the sample name text is segmented to obtain three first participles, namely [_G], [olf] and [_course]. Based on the order of their occurrence frequency from small to large, the first participles are sorted to obtain [_course][olf][_G]. Assuming that L is equal to 3, the first entity identifier is [_course][olf][_G].
[0241] In a possible implementation, the second image features, sample visual features, and sample position features are spliced and input into the first large language model for text prediction to generate a predicted probability distribution. Specifically, a prompt text is constructed to prompt the first large language model to generate an entity identifier; text features of the prompt text are extracted, and the second image features, text features, sample visual features, and sample position features are spliced and input into the first large language model for text prediction to generate a predicted probability distribution.
[0242] Specifically, the predicted probability distribution generated based on the first language model can determine the predicted identifier corresponding to each sample identifier, and all predicted identifiers can form a predicted entity identifier. The formula for determining the predicted identifier is as follows:
[0243]
[0244] in, The first The predicted identifiers for the locations, The first The predicted identifier before positions, is the text feature, is the second image feature, are sample visual features and sample position features, Used to indicate the word embedding matrix, Used to indicate the word embedding matrix Perform word embedding processing. The first The word embedding vector corresponding to the predicted identifier before the position in the word embedding matrix, Used to indicate the first largest language model, which can be based on the input , , as well as , generating the predicted entity identifier The probability distribution of the positions is then used to predict the predicted entity identifier. The predicted identifier of the position is then input into the first largest language model, so that the first largest language model can predict the predicted identifier of the next position, until the preset end condition is met, and all the predicted identifiers can form a predicted entity identifier.
[0245] The complete process of the identification generation method is described in detail below. The complete process includes a training phase and an inference phase.
[0246] The training phase is described in detail below.
[0247] Reference Fig.10 , Fig.10 A schematic diagram of an optional architecture of the training phase provided in an embodiment of the present disclosure.
[0248] First, a plurality of original images and query texts corresponding to the respective original images are obtained, and identification information corresponding to the respective original images is determined respectively according to the respective original images and the corresponding query texts, wherein the query texts are used to prompt the identification of the objects of interest in the corresponding original images; a plurality of candidate entities are obtained, and based on the respective identification information, the link entities corresponding to the respective original images are determined respectively among the plurality of candidate entities.
[0249] Then, each query text is input into the second largest language model for text prediction to generate a summary text corresponding to each original image; object detection is performed based on each original image and the corresponding summary text to generate an original bounding box corresponding to each original image, wherein the original bounding box is used to indicate the corresponding object of interest; each original image and the corresponding original bounding box is input into the first mask generation model for mask prediction to generate a labeled mask corresponding to each original image, wherein the labeled mask is used to indicate the corresponding object of interest.
[0250] Then, obtain the reference name text corresponding to each linked entity, wherein the reference name text is used to indicate the name of the reference entity, and the level of the reference entity in the knowledge base is higher than the level of the linked entity in the knowledge base; input each original image and the corresponding reference name text into the second mask generation model for mask prediction, and generate a reference mask corresponding to each original image; determine the degree of matching between each annotated mask and the corresponding reference mask, and obtain the target matching degree corresponding to each annotated mask; when the target matching degree is less than the preset matching degree threshold, eliminate the original image corresponding to the target matching degree.
[0251] Then, the number of connected areas in each annotation mask is counted to obtain the number of areas corresponding to each annotation mask; when the number of areas is greater than a preset threshold, the original image corresponding to the number of areas is eliminated; then, each original image, the corresponding annotation mask and the corresponding linked entity are associated and stored in a data set, wherein the sample image is sampled from multiple original images, the second mask is the annotation mask corresponding to the sample image, and the sample entity is the linked entity linked to the sample image.
[0252] Then, a sample image and a second mask corresponding to a sample object in the sample image are obtained, and the sample image is segmented to obtain a first mask corresponding to each visual object in the sample image, wherein the sample object is one of the multiple visual objects. For example, the sample image can be input into a target mask generation model for panoramic segmentation to obtain multiple first masks;
[0253] Then, extract the second image feature of the sample image. For example, the sample image can be input into a visual encoder for multi-level feature extraction to obtain multi-level coding features; the multi-level coding features are input into the first multi-layer perceptron for mapping to obtain the second image feature;
[0254] Then, feature extraction is performed on the sample image based on the second mask and each first mask to obtain a plurality of sample visual features, and sample position features corresponding to the second mask and each first mask are extracted. For example, the multi-level coding features, the second mask and each first mask can be input into a mask-aware visual extractor, and feature extraction is performed on the multi-level coding features based on the mask-aware visual extractor to obtain a plurality of sample visual features, and a plurality of sample position features corresponding to the second mask and each first mask are obtained through position coding.
[0255] Then, a prompt text is constructed to prompt the first language model to generate an entity identifier; text features of the prompt text are extracted; the second image features, text features of the prompt text, sample visual features, and sample position features are spliced and input into the first language model for text prediction to generate a prediction probability distribution, wherein the prediction probability distribution is used to determine the entity identifier of the sample object;
[0256] Then, a sample name text of a sample entity linked to the sample image is obtained, and the sample name text is segmented to obtain a plurality of first segmentations; the occurrence frequency of each first segmentation in the knowledge base is determined, and each first segmentation is sorted in order of the occurrence frequency from small to large, and the first segmentations arranged in the first L positions are determined as second segmentations, where L is a positive integer; based on each second segmentation, a first entity identifier of the sample entity is determined; a model loss is determined based on the predicted probability distribution and the first entity identifier, and a first large language model is trained based on the model loss, where the sample entity is used to indicate the sample object.
[0257] The inference phase is described in detail below.
[0258] First, a target image and a prompt mark of the target image are obtained, and the target image is segmented to obtain a local mask corresponding to each candidate object. The target image includes multiple candidate objects, and the prompt mark is used to indicate the selected target object among the multiple candidate objects.
[0259] Then, when the hint mark is a mark point, the query mask is determined in each local mask based on the positional relationship between the mark point and each local mask; or, when the hint mark is a mark box, the query mask is determined in each local mask based on the matching degree between the mark box and the mask boundary of each local mask; or, when the hint mark is a mark area, the query mask is determined in each local mask based on the matching degree between the mark area and each local mask. The query mask is used to indicate the target object selected from multiple candidate objects.
[0260] Then, multi-level feature extraction is performed on the target image to obtain multi-level visual features of the target image; mask pooling is performed on the multi-level visual features based on the query mask and each local mask to obtain multiple multi-level pooling features.
[0261] Then, for any multi-level pooling feature, the sub-features of each level in the multi-level pooling feature are mapped respectively to obtain multiple intermediate features of the same dimension, and the intermediate features are fused to obtain fused features; multi-layer perception processing is performed on each fused feature to obtain multiple mask visual features; and the mask position features corresponding to the query mask and each local mask are extracted.
[0262] Then, each mask visual feature is spliced with the corresponding mask position feature to obtain multiple regional features; the first image feature of the target image is extracted, the mask area of each local mask is determined respectively, and the regional features corresponding to each local mask are spliced in order of the size of the mask area to obtain the first splicing feature.
[0263] Then, a prompt text is constructed to prompt the first language model to generate an entity identifier; text features of the prompt text are extracted, and the first image features, text features, first splicing features, and regional features corresponding to the query mask are spliced to obtain target splicing features; text prediction is performed based on the target splicing features to generate a target entity identifier of the target object.
[0264] Based on this, by obtaining the local masks corresponding to each candidate object in the target image, and determining the query mask used to indicate the target object, and then obtaining the query mask and the mask visual features corresponding to each local mask through feature extraction, and extracting the query mask and the mask position features corresponding to each local mask, the mask visual features can capture the pixel-level visual information of the local area where the corresponding object is located, and the mask position features can capture the pixel-level position information of the local area where the corresponding object is located, and then each mask visual feature is spliced with the corresponding mask position features respectively to obtain the regional features of each local area, which is equivalent to combining the pixel-level visual information of the local area and the corresponding pixel-level position information into pixel-level regional information, and then extracting the first image of the target image. Features, and splice the first image features and multiple regional features into target splicing features, and then perform text prediction based on the target splicing features to generate a target entity identifier of the target object. In the text prediction process, by interacting with each feature in the target splicing feature, it is possible to pay attention to both the global visual information captured by the first image feature and the pixel-level regional information corresponding to each candidate object and the pixel-level regional information corresponding to the target object. Therefore, it is possible to effectively improve the understanding of the global image features and pixel-level details of the target image, thereby improving the prediction accuracy of the target entity identifier. In addition, using the pixel-level query mask as a visual cue can efficiently, flexibly and accurately refer to the target object, thereby further improving the prediction accuracy of the target entity identifier.
[0265] The training process of the model provided by the present invention is first described in detail below.
[0266] The first large language model provided by the present disclosure can be trained in two stages. In the first training stage, the first large language model is pre-trained using an initial sample set containing 2 million samples. In the second training stage, the pre-trained first large language model is fine-tuned by optimizing the entity subset and query subset in the sample set.
[0267] Since the optimized dataset contains about 4.5 million training samples, during the fine-tuning process, considering the limitation of computing resources and the huge size of the optimized dataset, the number of training samples corresponding to each entity can be limited to 50 or less, so that only about 7% of the training samples in the entity subset and query subset are used, with a total of about 300,000, which can effectively save computing resources and improve training efficiency. In addition, the size of all input images is uniformly preprocessed to 512×512 to ensure the consistency of training, and the length of entity identifiers is limited to 4 to avoid the length of entity identifiers being too long, thereby improving training efficiency.
[0268] Then, the evaluation process of the model provided by the present disclosure and other models is described in detail below.
[0269] The data set used for evaluation can be divided into a validation set and a test set. The evaluation results are shown in Table 3 below:
[0270] Table 3
[0271]
[0272] Among them, "None" means that no cue is used to refer to visual information, refers to the retrieval-based discriminant model, is the generative model, It refers to a zero-sample model that does not require fine-tuning. The accuracy of the column where "all" is located is the accuracy determined by the data set composed of the corresponding entity subset and the query subset. Compared with the model provided in the present disclosure, the reference model is different in that the visual features and position features of the mask corresponding to the object of interest in the image are input into the first largest language model, while the visual features and position features of the masks corresponding to other objects in the image are not input into the first largest language model. The reference model-fine-tuning refers to the reference model after fine-tuning.
[0273] Specifically, Table 3 shows the accuracy results of the vision-language model with different prompt types on the validation set and the test set. The accuracy is used as the key indicator to measure the performance of the model on the validation set and the test set. For each data set, the performance of the model on the entity subset and the query subset is evaluated, and the overall accuracy of all samples in each subset is calculated as the final evaluation basis.
[0274] In addition, considering the challenges of zero-shot inference models in generating entity identifiers and processing entity name texts in specific fields, BM25 retrieval is needed to process the generated results. Specifically, it is necessary to search 6 million entity name texts in the knowledge base and then select the most similar search results as the basis for calculating accuracy.
[0275] As can be seen from Table 3, compared with the visual language model based on text cues, the model provided by the present disclosure has achieved significant improvement on the entity subset, with the performance difference ranging from -2.0% to 11.3%. This evaluation result shows that the challenges brought about by the lack of text priors can be alleviated through refined visual feature modeling.
[0276] Then, compared to and , the performance of the model provided by the present disclosure on the query subset has a gap of 22% to 42%, which can be considered to be caused by the fact that the query subset mainly comes from visual question answering (VQA). The questions of visual question answering usually involve not only references to visual information, but also usually contain additional query intents, such as "made of...", "produced in", "how much water is needed", etc. These situations require text to express user intent and further reasoning, which is beyond the scope of VEL. The model provided by the present disclosure is a reference prompt based on a visual mask, so it cannot cover these problems well. Based on this, for the data in the query subset, it is necessary to use the second largest language model to extract its reference expression and replace it with a visual mask tag to retain the reference information other than the query intent.
[0277] Then, the ablation experiment of the model provided by the present disclosure is described in detail below.
[0278] The ablation experiment results are shown in Table 4 below:
[0279] Table 4
[0280]
[0281] Among them, PT refers to pre-training the model, and FT refers to fine-tuning the model. Compared with the model provided in the present disclosure, the reference model differs in that the visual features and position features of the mask corresponding to the object of interest in the image are input into the first large language model, while the visual features and position features of the masks corresponding to other objects in the image are not input into the first large language model. Therefore, the reference model can be regarded as a model in which the corresponding components for processing masks corresponding to other objects are removed from the model provided in the present disclosure.
[0282] Specifically, Table 4 shows the ablation experiment results of the model provided by the present disclosure, which can evaluate the effectiveness of visual semantic labeling and training. The experimental results show that the introduction of fine-grained local visual features can significantly improve the accuracy of the model, with the accuracy on the entity subset increased by 3.7% to 5.0%, and the accuracy on the query subset increased by 3.5% to 5.5%. In addition, fine-tuning also significantly improves the overall accuracy of the model, while the impact of pre-training is relatively small, with the improvement of pre-training ranging from 0.1% to 1.6%.
[0283] Based on this, it can be determined that the success of the model provided by the present disclosure in the pre-training stage is mainly due to: the construction of a large-scale pre-training data set, for example, the pre-training data set contains 55 million samples; and the use of Generative Image-to-text Transformer (GIT) as the backbone (with 400 million parameters) combined with a randomly initialized text decoder. By combining limited model parameters and the original pre-training strategy, it helps to provide the pre-training effect of the model provided by the present disclosure.
[0284] Then, the generalization ability of the model provided by the present disclosure is described in detail below.
[0285] Table 5
[0286]
[0287] in, refers to the retrieval-based discriminant model, is the generative model, It refers to a zero-sample model that does not require fine-tuning; "visible" means that the object of interest in the image is usually a visual entity, and "invisible" means that the object of interest in the image is usually not a visual entity; compared with the model provided in the present disclosure, the reference model is different in that the visual features and position features of the mask corresponding to the object of interest in the image are input into the first largest language model, while the visual features and position features of the masks corresponding to other objects in the image are not input into the first largest language model; reference model-fine-tuning refers to the reference model after fine-tuning.
[0288] Specifically, Table 5 shows the accuracy results of different models on the visible data subset and the invisible data subset. Since the model provided by the present disclosure lacks text priors, it is compared with the model based on text prompts when determining the generalization ability.
[0289] in, It is a VEL model based on text prompts. Candidate entities are retrieved using a CLIP-based encoder, and the final entity is generated through MLLM-based candidate prefix tree constraint decoding. These two stages are optimized end-to-end through a multi-task objective. Combined with search enhancement ( ) and decode to generate ( ),so The accuracy is relatively high on both the visible entity subset (about 30%) and the invisible entity subset (about 10%).
[0290] It is worth noting that Generated for no search enhancement and constraint decoding Model, The performance is very close to that of the model provided in the present invention, and the difference in accuracy ranges from -0.2% to +0.5%, which indicates that retrieval enhancement may be an effective way to improve the generalization ability of the model. Therefore, in the training process of the first language model, when it is necessary to determine the linked entities corresponding to each original image from multiple candidate entities, the retrieval enhancement method can be introduced to improve the generalization ability of the first language model.
[0291] It can be seen that the identification generation method provided by the embodiment of the present disclosure can be applied to various fields.
[0292] For example, in the field of smart transportation, road images can be collected through vehicle sensors, and the road images can be used as target images. Then, local masks corresponding to each candidate object in the target image are obtained, and a query mask is determined from multiple local masks, wherein the candidate objects may be pedestrians, vehicles, traffic signs, etc., and the query mask is used to indicate the target object selected from multiple candidate objects. For example, the driver or passenger can interact with the on-board system so that the on-board system can obtain a prompt mark of the target image, and the prompt mark is used to indicate the target object selected from multiple candidate objects; feature extraction is performed on the target image based on the query mask and each local mask to obtain multiple mask visual features, and the query mask and each local mask are extracted. corresponding mask position features; splicing each mask visual feature with the corresponding mask position feature to obtain multiple regional features; extracting the first image feature of the target image, splicing the first image feature and the multiple regional features to obtain the target splicing feature; performing text prediction based on the target splicing feature to generate a target entity identifier of the target object; being able to effectively improve the understanding of the global image features and pixel-level details of the target image, thereby improving the prediction accuracy of the target entity identifier; then, being able to accurately link the target object with the entity indicated by the target entity identifier to achieve visual entity linking, and then being able to query the associated information of the linked entity in the knowledge base, thereby helping the on-board system to more fully understand the information of the target object.
[0293] For another example, in the medical field, medical images can be collected by medical instruments. For example, medical images include magnetic resonance imaging images, pathological images, and computer tomography images, etc. Then, local masks corresponding to each candidate object in the target image are obtained, and a query mask is determined from multiple local masks, wherein the candidate objects may be organs, tissues, etc., and the query mask is used to indicate the target object selected from multiple candidate objects. For example, a doctor can interact with a medical system so that the medical system can obtain a prompt mark of the target image, and the prompt mark is used to indicate the target object selected from multiple candidate objects; feature extraction is performed on the target image based on the query mask and each local mask, respectively, to obtain multiple mask visual features, and the query mask and each local mask are extracted. mask position features corresponding to the local mask; splicing each mask visual feature with the corresponding mask position feature to obtain multiple regional features; extracting the first image feature of the target image, splicing the first image feature and multiple regional features to obtain the target splicing feature; performing text prediction based on the target splicing feature to generate the target entity identification of the target object; being able to effectively improve the understanding of the global image features and pixel-level details of the target image, thereby improving the prediction accuracy of the target entity identification; then, being able to accurately link the target object with the entity indicated by the target entity identification to achieve visual entity linking, and then being able to query the associated information of the linked entity in the knowledge base, thereby helping the medical system to more fully understand the information of the target object.
[0294] It is to be understood that, although the steps in the above-mentioned various flow charts are shown in sequence according to the indication of the arrows, these steps are not necessarily performed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in the present embodiment, the execution of these steps does not have a strict order restriction, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the above-mentioned flow charts can include a plurality of steps or a plurality of stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of the steps or stages in other steps or other steps.
[0295] Reference Fig.11 , Fig.11 This is an optional structural diagram of an identification generation device provided in an embodiment of the present disclosure. The identification generation device 1100 includes:
[0296] An acquisition module 1101 is used to acquire a local mask corresponding to each candidate object in a target image, and determine a query mask from a plurality of local masks, wherein the query mask is used to indicate a target object selected from a plurality of candidate objects;
[0297] The feature extraction module 1102 is used to perform feature extraction on the target image based on the query mask and each local mask, obtain multiple mask visual features, and extract mask position features corresponding to the query mask and each local mask;
[0298] A first splicing module 1103 is used to splice each mask visual feature with the corresponding mask position feature to obtain multiple regional features;
[0299] The second stitching module 1104 is used to extract the first image feature of the target image, and stitch the first image feature and multiple regional features to obtain a target stitching feature;
[0300] The generation module 1105 is used to perform text prediction based on the target splicing features and generate a target entity identifier of the target object.
[0301] Furthermore, the second splicing module 1104 is specifically used for:
[0302] Determine the mask area of each local mask respectively, and splice the regional features corresponding to each local mask in order of the size of the mask area to obtain a first splicing feature;
[0303] The first image feature, the first stitching feature, and the regional feature corresponding to the query mask are stitched together to obtain a target stitching feature.
[0304] Furthermore, the second splicing module 1104 is specifically used for:
[0305] Constructing a prompt text for prompting the first language model to generate entity identification;
[0306] The text feature of the prompt text is extracted, and the first image feature, the text feature, the first splicing feature and the regional feature corresponding to the query mask are spliced to obtain the target splicing feature.
[0307] Furthermore, the feature extraction module 1102 is specifically used for:
[0308] Perform multi-level feature extraction on the target image to obtain multi-level visual features of the target image;
[0309] Based on the query mask and each local mask, the multi-level visual features are masked and pooled to obtain multiple multi-level pooled features;
[0310] Each multi-level pooling feature is fused separately to obtain multiple mask visual features.
[0311] Furthermore, the feature extraction module 1102 is specifically used for:
[0312] For any multi-level pooling feature, the sub-features of each level in the multi-level pooling feature are mapped respectively to obtain multiple intermediate features with the same dimensions, and the intermediate features are fused to obtain the fused feature;
[0313] Multi-layer perception processing is performed on each fusion feature to obtain multiple mask visual features.
[0314] Furthermore, the acquisition module 1101 is specifically used for:
[0315] Acquire a target image and a prompt mark of the target image, wherein the target image includes a plurality of candidate objects, and the prompt mark is used to indicate a target object selected from the plurality of candidate objects;
[0316] The target image is segmented to obtain local masks corresponding to each candidate object, and the query mask is determined in each local mask based on the hint mark.
[0317] Furthermore, the acquisition module 1101 is specifically used for:
[0318] When the hint mark is a marked point, the query mask is determined in each local mask based on the positional relationship between the marked point and each local mask;
[0319] Alternatively, when the hint is marked as a marked box, the query mask is determined in each local mask based on a degree of matching between the marked box and a mask boundary of each local mask;
[0320] Alternatively, when the hint is marked as a labeled region, the query mask is determined in each local mask based on the matching degree between the labeled region and each local mask.
[0321] Furthermore, the target entity identifier is generated by the first language model. The identifier generation device further includes a training module (not shown in the figure). The training module is specifically used for:
[0322] Acquire a sample image and a second mask corresponding to a sample object in the sample image, segment the sample image, and obtain a first mask corresponding to each visual object in the sample image, wherein the sample object is one of the multiple visual objects;
[0323] Extracting a second image feature of the sample image, performing feature extraction based on the second mask and each of the first masks on the sample image to obtain a plurality of sample visual features, and extracting sample position features corresponding to the second mask and each of the first masks;
[0324] splicing the second image feature, the sample visual feature, and the sample position feature and inputting them into the first language model for text prediction to generate a prediction probability distribution, wherein the prediction probability distribution is used to determine the entity identifier of the sample object;
[0325] A first entity identifier of a sample entity linked to the sample image is obtained, a model loss is determined based on the predicted probability distribution and the first entity identifier, and a first language model is trained based on the model loss, wherein the sample entity is used to indicate a sample object.
[0326] Furthermore, the sample image, the second mask and the first entity identifier are all obtained from the data set, and the training module is also used for:
[0327] Acquire multiple original images and query texts corresponding to each original image, and determine identification information corresponding to each original image according to each original image and the corresponding query text, wherein the query text is used to prompt identification of the object of interest in the corresponding original image;
[0328] Acquire multiple candidate entities, and determine link entities corresponding to each original image from the multiple candidate entities based on each piece of identification information;
[0329] Based on each original image and the corresponding query text, respectively determine a labeling mask for each original image, wherein the labeling mask is used to indicate the corresponding object of interest;
[0330] Each original image, the corresponding annotation mask and the corresponding link entity are associated and stored in a data set, wherein the sample image is sampled from multiple original images, the second mask is the annotation mask corresponding to the sample image, and the sample entity is the link entity linked to the sample image.
[0331] Furthermore, the above training module is specifically used for:
[0332] Input each query text into the second largest language model for text prediction to generate summary text corresponding to each original image;
[0333] Performing object detection based on each original image and the corresponding summarized text, respectively, to generate an original bounding box corresponding to each original image, wherein the original bounding box is used to indicate the corresponding object of interest;
[0334] Each original image and the corresponding original bounding box are respectively input into the first mask generation model for mask prediction to generate a labeled mask corresponding to each original image.
[0335] Furthermore, the above training module is also used for:
[0336] Obtaining a reference name text corresponding to each link entity, wherein the reference name text is used to indicate the name of the reference entity, and the level of the reference entity in the knowledge base is higher than the level of the link entity in the knowledge base;
[0337] Inputting each original image and the corresponding reference name text into the second mask generation model for mask prediction to generate a reference mask corresponding to each original image;
[0338] Determine the matching degree between each annotation mask and the corresponding reference mask, and obtain the target matching degree corresponding to each annotation mask;
[0339] When the target matching degree is less than a preset matching degree threshold, the original image corresponding to the target matching degree is discarded.
[0340] Furthermore, the above training module is also used for:
[0341] Count the number of connected areas in each annotation mask to obtain the number of areas corresponding to each annotation mask;
[0342] When the number of regions is greater than a preset threshold, the original image corresponding to the number of regions is discarded.
[0343] Furthermore, the above training module is specifically used for:
[0344] Obtaining a sample name text of a sample entity linked to the sample image, and performing word segmentation on the sample name text to obtain a plurality of first word segmentations;
[0345] Determine the occurrence frequency of each first participle in the knowledge base, sort each first participle based on the order of the occurrence frequency from small to large, and determine the first participle arranged in the first L positions as the second participle, where L is a positive integer;
[0346] Based on each second participle, a first entity identifier of the sample entity is determined.
[0347] The above-mentioned identification generation device 1100 and the identification generation method are based on the same inventive concept, by obtaining the local mask corresponding to each candidate object in the target image, and determining the query mask used to indicate the target object, and then obtaining the query mask and the mask visual features corresponding to each local mask through feature extraction, and extracting the query mask and the mask position features corresponding to each local mask, the mask visual features can capture the pixel-level visual information of the local area where the corresponding object is located, the mask position features can capture the pixel-level position information of the local area where the corresponding object is located, and then each mask visual feature is spliced with the corresponding mask position features respectively to obtain the regional features of each local area, which is equivalent to combining the pixel-level visual information of the local area and the corresponding pixel-level position information into pixel-level regional information, and then The first image feature of the target image is then extracted, and the first image feature and multiple regional features are spliced into a target splicing feature. Then, text prediction is performed based on the target splicing feature to generate a target entity identifier of the target object. In the text prediction process, by interacting with each feature in the target splicing feature, it is possible to pay attention to both the global visual information captured by the first image feature and the pixel-level regional information corresponding to each candidate object and the pixel-level regional information corresponding to the target object. Therefore, it is possible to effectively improve the understanding of the global image features and pixel-level details of the target image, thereby improving the prediction accuracy of the target entity identifier. In addition, using the pixel-level query mask as a visual cue can efficiently, flexibly and accurately refer to the target object, thereby further improving the prediction accuracy of the target entity identifier.
[0348] The electronic device for executing the above-mentioned identification generation method provided in the embodiment of the present disclosure may be a terminal. Fig.12 , Fig.12 This is a partial structural block diagram of a terminal provided in an embodiment of the present disclosure, and the terminal includes: a camera assembly 1210, a first memory 1220, an input unit 1230, a display unit 1240, a sensor 1250, an audio circuit 1260, a wireless fidelity (WiFi) module 1270, a first processor 1280, and a first power supply 1290. Those skilled in the art can understand that Fig.12 The terminal structure shown in the figure does not constitute a limitation on the terminal, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.
[0349] The camera assembly 1210 can be used to capture images or videos. Optionally, the camera assembly 1210 includes a front camera and a rear camera. Typically, the front camera is disposed on the front panel of the terminal, and the rear camera is disposed on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize the panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions.
[0350] The first memory 1220 may be used to store software programs and modules. The first processor 1280 executes various functional applications and data processing of the terminal by running the software programs and modules stored in the first memory 1220 .
[0351] The input unit 1230 may be used to receive input digital or character information and generate key signal input related to the terminal's settings and function control. Specifically, the input unit 1230 may include a touch panel 1231 and other input devices 1232 .
[0352] The display unit 1240 may be used to display input information or provided information and various menus of the terminal. The display unit 1240 may include a display panel 1241 .
[0353] The audio circuit 1260 , the speaker 1261 , and the microphone 1262 may provide an audio interface.
[0354] The first power source 1290 may be alternating current, direct current, a disposable battery, or a rechargeable battery.
[0355] The number of sensors 1250 may be one or more, and the one or more sensors 1250 include but are not limited to: acceleration sensors, gyroscope sensors, pressure sensors, optical sensors, etc. Among them:
[0356] The acceleration sensor can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established by the terminal. For example, the acceleration sensor can be used to detect the components of gravity acceleration on the three coordinate axes. The first processor 1280 can control the display unit 1240 to display the user interface in a horizontal view or a vertical view according to the gravity acceleration signal collected by the acceleration sensor. The acceleration sensor can also be used for collecting game or user motion data.
[0357] The gyroscope sensor can detect the body direction and rotation angle of the terminal, and the gyroscope sensor can cooperate with the acceleration sensor to collect the user's 3D actions on the terminal. The first processor 1280 can implement the following functions based on the data collected by the gyroscope sensor: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.
[0358] The pressure sensor can be set in the side frame of the terminal and / or the lower layer of the display unit 1240. When the pressure sensor is set in the side frame of the terminal, the user's holding signal of the terminal can be detected, and the first processor 1280 performs left and right hand recognition or shortcut operation according to the holding signal collected by the pressure sensor. When the pressure sensor is set in the lower layer of the display unit 1240, the first processor 1280 controls the operability controls on the UI interface according to the user's pressure operation on the display unit 1240. The operability control includes at least one of a button control, a scroll bar control, an icon control, and a menu control.
[0359] The optical sensor is used to collect the ambient light intensity. In one embodiment, the first processor 1280 can control the display brightness of the display unit 1240 according to the ambient light intensity collected by the optical sensor. Specifically, when the ambient light intensity is high, the display brightness of the display unit 1240 is increased; when the ambient light intensity is low, the display brightness of the display unit 1240 is decreased. In another embodiment, the first processor 1280 can also dynamically adjust the shooting parameters of the camera assembly 1210 according to the ambient light intensity collected by the optical sensor.
[0360] In this embodiment, the first processor 1280 included in the terminal can execute the identifier generation method of the previous embodiment.
[0361] The electronic device for executing the above-mentioned identification generation method provided in the embodiment of the present disclosure may also be a server. Fig.13 , Fig.13 This is a partial structural block diagram of a server provided in an embodiment of the present disclosure. The server may have relatively large differences due to different configurations or performances, and may include one or more second processors 1310 and a second memory 1330, and one or more storage media 1340 (e.g., one or more mass storage devices) storing application programs 1343 or data 1342. Among them, the second memory 1330 and the storage medium 1340 may be short-term storage or permanent storage. The program stored in the storage medium 1340 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Furthermore, the second processor 1310 may be configured to communicate with the storage medium 1340 to execute a series of instruction operations in the storage medium 1340 on the server.
[0362] The server may also include one or more second power supplies 1320, one or more wired or wireless network interfaces 1350, one or more input and output interfaces 1360, and / or one or more operating systems 1341, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.
[0363] The second processor 1310 in the server may be configured to execute the identification generating method.
[0364] The embodiments of the present disclosure further provide a computer-readable storage medium, which is used to store a computer program, and the computer program is used to execute the identification generation method of each of the aforementioned embodiments.
[0365] The embodiment of the present disclosure also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the above-mentioned identification generation method.
[0366] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present disclosure described herein can, for example, be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0367] It should be understood that in the present disclosure, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.
[0368] It should be understood that in the description of the embodiments of the present disclosure, the meaning of multiple (or multiple items) is more than two, greater than, less than, exceed, etc. are understood as not including the number, and above, below, within, etc. are understood as including the number.
[0369] In the several embodiments provided in the present disclosure, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.
[0370] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0371] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0372] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present disclosure. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store program codes.
[0373] It should also be understood that the various implementations provided in the embodiments of the present disclosure can be combined arbitrarily to achieve different technical effects.
[0374] The above is a specific description of the preferred implementation of the present disclosure, but the present disclosure is not limited to the above-mentioned implementation mode. Technical personnel familiar with the field can also make various equivalent deformations or substitutions under the shared conditions without violating the spirit of the present disclosure. These equivalent deformations or substitutions are all included in the scope defined by the claims of the present disclosure.
Claims
1. A method for generating an identifier, characterized in that: include: Acquire a local mask corresponding to each candidate object in the target image, and determine a query mask from the multiple local masks, wherein the query mask is used to indicate a target object selected from the multiple candidate objects; Performing feature extraction based on the query mask and each of the local masks on the target image to obtain a plurality of mask visual features, flattening the query mask and each of the local masks and inputting them into a position encoder for mapping to obtain mask position features corresponding to the query mask and each of the local masks; splicing each of the mask visual features with the corresponding mask position features to obtain a plurality of regional features; Extracting a first image feature of the target image, and splicing the first image feature and a plurality of the regional features to obtain a target splicing feature; Text prediction is performed based on the target concatenation features to generate a target entity identifier of the target object.
2. The identification generation method according to claim 1, characterized in that: The step of splicing the first image feature and the plurality of regional features to obtain a target splicing feature includes: Determine the mask area of each of the local masks respectively, and splice the regional features corresponding to each of the local masks in order of the size of the mask area to obtain a first splicing feature; The first image feature, the first stitching feature, and the region feature corresponding to the query mask are stitched together to obtain a target stitching feature.
3. The identification generation method according to claim 2, characterized in that: The target entity identifier is generated by a first large language model, and the first image feature, the first splicing feature, and the region feature corresponding to the query mask are spliced to obtain a target splicing feature, including: Constructing a prompt text for prompting the first language model to generate an entity identifier; The text feature of the prompt text is extracted, and the first image feature, the text feature, the first splicing feature and the region feature corresponding to the query mask are spliced to obtain a target splicing feature.
4. The identification generation method according to claim 1, characterized in that: The extracting features of the target image based on the query mask and each of the local masks to obtain a plurality of mask visual features includes: Performing multi-level feature extraction on the target image to obtain multi-level visual features of the target image; Based on the query mask and each of the local masks, respectively, mask pooling is performed on the multi-level visual features to obtain a plurality of multi-level pooled features; Feature fusion is performed on each of the multi-level pooling features to obtain multiple mask visual features.
5. The identification generation method according to claim 4, characterized in that: The step of fusing the multi-level pooling features to obtain a plurality of mask visual features comprises: For any of the multi-level pooling features, sub-features of each level in the multi-level pooling features are mapped respectively to obtain multiple intermediate features of the same dimension, and each of the intermediate features is subjected to feature fusion to obtain a fused feature; Multi-layer perception processing is performed on each of the fused features to obtain multiple mask visual features.
6. The identification generation method according to claim 1, characterized in that: The acquiring of local masks corresponding to the candidate objects in the target image and determining a query mask from a plurality of the local masks comprises: Acquire a target image and a prompt mark of the target image, wherein the target image includes a plurality of candidate objects, and the prompt mark is used to indicate a target object selected from the plurality of candidate objects; The target image is segmented to obtain local masks corresponding to the candidate objects, and a query mask is determined in each of the local masks based on the prompt mark.
7. The identification generation method according to claim 6, characterized in that: The step of determining a query mask in each of the local masks based on the hint mark comprises: When the hint mark is a mark point, determining a query mask in each of the local masks based on a positional relationship between the mark point and each of the local masks; Alternatively, when the hint mark is a mark box, determining the query mask in each of the local masks based on a matching degree between the mark box and a mask boundary of each of the local masks; Alternatively, when the hint mark is a marked area, a query mask is determined in each of the local masks based on a matching degree between the marked area and each of the local masks.
8. The identification generation method according to claim 1, characterized in that: The target entity identifier is generated by a first language model, and the first language model is trained by the following steps: Acquire a sample image and a second mask corresponding to a sample object in the sample image, segment the sample image, and obtain a first mask corresponding to each visual object in the sample image, wherein the sample object is one of the multiple visual objects; Extracting a second image feature of the sample image, performing feature extraction based on the second mask and each of the first masks on the sample image to obtain a plurality of sample visual features, and extracting sample position features corresponding to the second mask and each of the first masks; splicing the second image feature, the sample visual feature, and the sample position feature and inputting them into the first language model for text prediction to generate a prediction probability distribution, wherein the prediction probability distribution is used to determine the entity identifier of the sample object; Obtain a first entity identifier of a sample entity linked to the sample image, determine a model loss based on the predicted probability distribution and the first entity identifier, and train the first large language model based on the model loss, wherein the sample entity is used to indicate the sample object.
9. The identification generation method according to claim 8, characterized in that: The sample image, the second mask, and the first entity identifier are all obtained from a data set. Before obtaining the sample image and the second mask corresponding to the sample object in the sample image, the identifier generation method further includes: Acquire multiple original images and query texts corresponding to each of the original images, and determine identification information corresponding to each of the original images according to each of the original images and the corresponding query texts, wherein the query texts are used to prompt identification of the object of interest in the corresponding original image; Acquire a plurality of candidate entities, and determine, based on the respective identification information, link entities corresponding to the respective original images from the plurality of candidate entities; Based on each of the original images and the corresponding query text, respectively determine a labeling mask for each of the original images, wherein the labeling mask is used to indicate the corresponding object of interest; Each of the original images, the corresponding annotation mask, and the corresponding linked entity are associated and stored in the data set, wherein the sample image is sampled from multiple original images, the second mask is the annotation mask corresponding to the sample image, and the sample entity is the linked entity linked to the sample image.
10. The identification generation method according to claim 9, characterized in that: The step of determining the annotation mask of each of the original images based on the original images and the corresponding query texts includes: Inputting each of the query texts into the second largest language model for text prediction to generate summary texts corresponding to each of the original images; Performing object detection based on each of the original images and the corresponding summarized text, respectively, to generate an original bounding box corresponding to each of the original images, wherein the original bounding box is used to indicate the corresponding object of interest; Each of the original images and the corresponding original bounding boxes are respectively input into a first mask generation model for mask prediction to generate a labeled mask corresponding to each of the original images.
11. The identification generation method according to claim 10, characterized in that: After inputting each of the original images and the corresponding original bounding boxes into the first mask generation model for mask prediction and generating a label mask corresponding to each of the original images, the identification generation method further includes: Acquire a reference name text corresponding to each of the link entities, wherein the reference name text is used to indicate the name of the reference entity, and the level of the reference entity in the knowledge base is higher than the level of the link entity in the knowledge base; Inputting each of the original images and the corresponding reference name text into a second mask generation model for mask prediction to generate a reference mask corresponding to each of the original images; Determining the matching degree between each of the annotated masks and the corresponding reference mask to obtain the target matching degree corresponding to each of the annotated masks; When the target matching degree is less than a preset matching degree threshold, the original image corresponding to the target matching degree is discarded.
12. The identification generation method according to claim 9, characterized in that: Before associating and storing each of the original images, the corresponding annotation masks, and the corresponding link entities in the data set, the identification generation method further includes: Counting the number of connected regions in each of the annotation masks to obtain the number of regions corresponding to each of the annotation masks; When the number of regions is greater than a preset number threshold, the original image corresponding to the number of regions is discarded.
13. The identification generation method according to claim 8, characterized in that: The obtaining of the first entity identifier of the sample entity linked to the sample image includes: Acquire a sample name text of a sample entity linked to the sample image, and segment the sample name text to obtain a plurality of first segmented words; Determine the occurrence frequency of each of the first participles in the knowledge base, sort each of the first participles based on the order of the occurrence frequency from small to large, and determine the first participles arranged in the first L positions as the second participles, where L is a positive integer; Based on each of the second participles, a first entity identifier of the sample entity is determined.
14. A device for generating an identification, characterized in that: include: An acquisition module, configured to acquire a local mask corresponding to each candidate object in a target image, and determine a query mask from a plurality of the local masks, wherein the query mask is used to indicate a target object selected from the plurality of candidate objects; A feature extraction module, configured to extract features of the target image based on the query mask and each of the local masks, obtain a plurality of mask visual features, flatten the query mask and each of the local masks, and input the flattened features into a position encoder for mapping, to obtain mask position features corresponding to the query mask and each of the local masks; A first splicing module, used for splicing each of the mask visual features with the corresponding mask position features to obtain a plurality of regional features; A second splicing module is used to extract a first image feature of the target image, and splice the first image feature and a plurality of the regional features to obtain a target splicing feature; A generation module is used to perform text prediction based on the target splicing features and generate a target entity identifier of the target object.
15. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the identification generation method described in any one of claims 1 to 13 is implemented.
16. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the identification generation method according to any one of claims 1 to 13 is implemented.
17. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the identification generation method according to any one of claims 1 to 13 is implemented.
