A multi-scale visual positioning method and system based on semantic consistency guidance

Abstract

The application relates to the technical field of visual language fusion positioning, and discloses a multi-scale visual positioning method and system based on semantic consistency guidance, which comprises the following steps: extracting a noun phrase based on a Stanza syntax analysis model, generating text semantic features, phrase semantic features and visual features respectively based on a pre-trained BEiT-3 image-text encoding model; calculating a text consistency constraint loss through the attention interaction of the noun phrase features and the visual features, and simultaneously generating a concept-level semantic heat map; based on the semantic heat map, applying weights corresponding to semantic responses to multi-scale visual features generated by a feature pyramid network; and according to the multi-scale visual features generated after weighting, carrying out adaptive sampling based on an offset amount on a candidate region through a deformable attention module and completing multi-scale candidate frame generation; the application solves the problem that the candidate region generation of an existing model lacks semantic directionality, and improves the problems of feature redundancy and unstable semantic alignment.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of visual language fusion positioning, and in particular to a multi-scale visual positioning method and system based on semantic consistency guidance. BACKGROUND

[0002] Visual language positioning is a cross-modal understanding task aiming to locate the target region corresponding to the semantic content in the image according to the text description. The existing methods usually adopt a region proposal mechanism relying only on image visual features in the candidate region generation stage, and generate a set of potential regions through a general object detection structure or multi-scale visual features. However, such methods do not introduce text semantics into the candidate region generation process in advance, lack structured analysis and semantic guidance of the description content, and the candidate regions are completely dominated by visual saliency, and cannot reflect the entity, attribute or relationship information in the text. In complex scenes, due to the large number of background components, significant target occlusion or dense similar objects, pure visual-driven region proposal often produces a large number of redundant regions irrelevant to the text description. These redundant regions not only increase the search space of subsequent cross-modal matching, but also cause noise interference in the semantic alignment process. It can be seen that the current visual language positioning model still has the problems of feature redundancy, unstable semantic alignment and inaccurate positioning in the candidate region generation stage. SUMMARY

[0003] The present application provides a multi-scale visual positioning method and system based on semantic consistency guidance to solve the problem of lack of semantic directionality in the candidate region generation of existing models, and improve the problems of feature redundancy, unstable semantic alignment and insufficient positioning accuracy.

[0004] In order to achieve the above purpose, the technical scheme is as follows:

[0005] In a first aspect, the present application provides a multi-scale visual positioning method based on semantic consistency guidance, comprising:

[0006] Obtaining the text description corresponding to the input image, after formatting processing of the text content, using the Stanza syntax analysis model to perform syntax dependency analysis on the input text description, identifying the syntax boundary and composition structure of the noun phrase in the sentence, and extracting the text content corresponding to the noun phrase;

[0007] Based on the pre-trained BEiT-3 image-text encoding model, the input image, text description and noun phrase are multi-modal encoded, the input image generates visual features, the text description and noun phrase generate overall text semantic features and noun phrase semantic features, respectively;

[0008] The text consistency constraint loss is calculated through the attention interaction between the noun phrase feature and the visual feature. When cross-attention calculation is performed, the attention weight of the noun phrase on each visual region and the response distribution are calculated, and a concept-level semantic heat map is generated.

[0009] The visual feature generates multi-scale visual features through a feature pyramid network, and applies corresponding weights to each scale visual feature according to the response distribution in the semantic heat map. The spatial intensity of the heat map is mapped into a weighting factor and acts on the feature maps of different scales. The spatial position and channel response of the multi-scale features are subjected to semantic-guided weighted processing.

[0010] Based on the multi-scale visual feature and the semantic heat map, a candidate region generation mechanism is constructed. The semantic related regions are enhanced through weighted features, and the object nature prediction and bounding box regression are completed by the candidate region generation head, so as to obtain a multi-scale candidate region set highly related to the text description.

[0011] The trained candidate box generation model is used to directly generate candidate region boxes and their corresponding visual features for test set images. Then, cross-modal similarity calculation is performed between these candidate box visual features and the semantic features of the text description. According to the similarity size, all candidate boxes are sorted, and the candidate box closest to the text semantics is selected as the final output to complete the positioning task.

[0012] Optionally, the text description corresponding to the input image is obtained, and the text is preprocessed to standardize the text format. The preprocessed text description is subjected to syntax dependency analysis based on a Stanza syntax analysis model to identify the syntax boundary and composition structure of the noun phrase in the sentence, and the text content corresponding to the noun phrase is extracted, including:

[0013] The input image corresponding to the text description is preprocessed to standardize the text format, and the preprocessed text is input into the Stanza syntax analysis model to perform syntax dependency analysis on the text description to extract the noun phrase and related syntax relationship in the sentence, and obtain the noun phrase .

[0014] The input image and the text description are both from the training set of CIR and LaSCo datasets.

[0015] Optionally, the input image, the text description, and the noun phrase are multi-modal encoded. The input image generates visual features, the text description and the noun phrase generate overall text semantic features and noun phrase semantic features, respectively, including:

[0016] The input image text description and noun phrase input into the picture-text coding model to extract visual features , text semantic features and phrase semantic features .

[0017] Optionally, the text consistency constraint loss is calculated through the attention interaction between the noun phrase features and the visual features. When cross-attention calculation is performed, the attention weight of the noun phrase on each visual region and the response distribution are calculated, and a concept-level semantic heat map is generated, including:

[0018] By embedding each noun phrase into its corresponding position in the original sentence and preserving its surrounding dependency context, a context version text segment is constructed, and the visual feature set and the context version text segment are input into the cross-modal attention module to obtain a context attention map, and the calculation process satisfies the following relationship:

[0019] , , ;

[0020] ;

[0021] In the formula, , , respectively represent the query, key, and value of the noun phrase under the condition of the context version text segment, represents the i-th noun phrase semantic feature, represents the context version text segment corresponding to the i-th noun phrase, represents the cross-modal attention map of the i-th noun phrase under the condition of the context version text segment, , , respectively represent three fully connected layers of the noun phrase under the condition of the context version text segment, represents the activation function, represents the dimension of the text feature;

[0022] The visual feature set and the noun phrase are input into the cross-modal attention module to obtain an independent phrase attention map, and the calculation process satisfies the following relationship:

[0023] , , ;

[0024] ;

[0025] wherein, , , respectively represent the query, key, value when the i-th noun phrase is independently input, represent the cross-modal attention map when the i-th noun phrase is independently input, , , respectively represent three full connection layers of the noun phrase under the independent condition;

[0026] The context attention map and the independent phrase attention map are differentially processed and a consistency difference matrix is generated through a non-negative operation. Subsequently, all consistency difference matrices are accumulated according to the noun phrase index to form a text consistency constraint loss, and the calculation process satisfies the following relationship:

[0027] ;

[0028] ;

[0029] wherein, represents the difference matrix of the i-th noun phrase, represents an activation function, represents a threshold value, represents a text consistency loss function;

[0030] Finally, a concept-level semantic heat map is generated based on the spatial response of the context attention map, and the generation process satisfies the following relationship:

[0031] ;

[0032] wherein, represents a heat map, represents an attention map merging function.

[0033] Optionally, the visual features are generated into multi-scale visual features through a feature pyramid network, including:

[0034] The feature pyramid network first performs down-sampling processing on the visual features at different magnifications, and then applies convolution mapping to the down-sampled features to obtain a multi-level visual feature set, and the sampling process satisfies the following relationship:

[0035] ;

[0036] , ;

[0037] wherein, represents a multi-level visual feature map set, a visual feature map of an s-th scale, a constructed convolution function of the s-th scale, down-sampling the visual feature at different scales, a number of levels of the multi-scale visual feature map.

[0038] Optionally, according to the response distribution in the semantic heat map, corresponding weights are applied to the visual features of each scale, the spatial intensity of the heat map is mapped to a weighting factor and is applied to the feature maps of different scales, including:

[0039] The semantic heat map is aligned according to the spatial size of the feature map of each scale to obtain a heat map representation consistent with the corresponding scale, and the alignment process satisfies the following relationship:

[0040] ;

[0041] In the formula, a heat map of the s-th scale visual feature after alignment, a scale transformation function, a spatial scale;

[0042] The aligned heat map is convolved and mapped and nonlinearly transformed to generate a weight matrix for adjusting the intensity of the visual feature, and the generation process satisfies the following relationship:

[0043] ;

[0044] In the formula, a 1x1 scale convolution function, a hyperparameter, an activation function, a weighting matrix of the s-th scale;

[0045] Finally, the weight matrix and the visual feature of the corresponding scale are point-by-point multiplied to apply different semantic weights to the visual responses of different spatial positions, realize cross-scale semantic guided feature enhancement, and the weighted scale visual feature is:

[0046] ;

[0047] In the formula, a weighted s-th scale visual feature, an element-by-element multiplication.

[0048] Optionally, a candidate region generation mechanism is constructed based on the multi-scale visual feature and the semantic heat map, the semantic related region is enhanced by the weighted feature, and the object nature prediction and the bounding box regression are completed by the candidate region generation head, including:

[0049] In each scale s, the weighted visual feature is input into the objectness prediction module to obtain the objectness prediction value, which is then converted into the objectness confidence, and the calculation process satisfies the following relationship:

[0050] ;

[0051] ;

[0052] In the formula, and respectively represent the axis coordinates and axis coordinates of the feature map, represents the objectness prediction value at the s-th scale position , represents the classification convolution function of the s-th scale, represents the visual feature at the s-th scale position after weighting, represents the objectness confidence at the s-th scale position ;

[0053] The boundary regression module is introduced to predict the offset amount of the four directions of the candidate frame, and the offset regression process satisfies the following relationship:

[0054] ;

[0055] In the formula, represents the boundary offset vector at the s-th scale position , represents the regression convolution function of the s-th scale, , , , respectively represent the distance values of the s-th scale position to the left boundary, the upper boundary, the right boundary, and the lower boundary of the candidate region;

[0056] The grid position on the feature map is mapped back to the original image coordinates to generate a candidate boundary frame center point with actual physical meaning, and the mapping relationship satisfies the following relationship:

[0057] ;

[0058] In the formula, represents the down-sampling step of the s-th scale, represents the center coordinates of the original picture corresponding to the s-th scale position ;

[0059] The geometric relationship can be used to construct the upper left corner and the lower right corner coordinates of the candidate box in the original image, thereby forming a standard rectangular bounding box representation, and the construction process satisfies the following relationship:

[0060] ;

[0061] ;

[0062] ;

[0063] ;

[0064] In the formula, and represent the x-axis coordinate and y-axis coordinate of the upper left corner of the candidate box, and represent the x-axis coordinate and y-axis coordinate of the lower right corner of the candidate box;

[0065] For each candidate position, a comprehensive confidence score is constructed to reflect the information of "whether the region is an object" and "whether the region is related to the text semantics". Then, all candidate box sets are filtered according to the comprehensive score, and the calculation and filtering process satisfies the following relationship:

[0066] ;

[0067] ;

[0068] In the formula, represents the comprehensive score of the s-th scale position , represents a hyperparameter, represents the visual feature aligned heat map of the s-th scale position , represents the filtered candidate box set of the s-th scale position , represents the score threshold.

[0069] Optionally, the method further comprises:

[0070] Using the trained candidate box generation model to directly generate candidate region boxes and their corresponding visual features for the test set images; then performing cross-modal similarity calculation on the candidate box visual features and the semantic features of the text description; sorting all candidate boxes according to the similarity size, and selecting the candidate box closest to the text semantics as the final output to complete the positioning task.

[0071] In a second aspect, the embodiments of the present application provide a multi-scale visual positioning system based on semantic consistency guidance, comprising a processor, a memory;

[0072] a memory for storing a computer program;

[0073] a processor for executing the program stored on the memory, realizing the method of any one of the first aspect.

[0074] Advantages:

[0075] The multi-scale visual positioning method based on semantic consistency guidance provided by the present application introduces the text consistency association between the semantic features of noun phrases and visual features, and uses the concept-level semantic heat map to guide the weighting of multi-scale visual features, thereby injecting text semantic constraints in the candidate region generation stage, effectively improving the problems of existing visual language positioning models that rely only on image generation, lack of semantic directionality, and unstable semantic alignment.

[0076] At the same time, by introducing an adaptive sampling mechanism of deformable attention in the candidate box generation process, the key visual information at different scales is dynamically integrated, the fine-grained matching ability of the candidate region and the text semantics is improved, the problems of candidate region redundancy, insufficient positioning accuracy and unstable cross-modal matching in complex scenes are effectively improved, and significant performance improvement is achieved on multiple benchmark datasets, improving the robustness and stability of the model. BRIEF DESCRIPTION OF DRAWINGS

[0077] Figure 1 The flowchart of the multi-scale visual positioning based on semantic consistency guidance of the embodiments of the present application;

[0078] Figure 2 The logic diagram of the multi-scale visual positioning based on semantic consistency guidance of the embodiments of the present application; DETAILED DESCRIPTION

[0079] The technical solutions of the present application will be described below in a clear and complete manner. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0080] Unless otherwise defined, technical terms and scientific terms used in the present application shall have the meanings as understood by a person having ordinary skill in the art to which the present application pertains. The terms "first", "second", and similar terms are used herein merely to distinguish one element from another, and are not intended to imply any order or sequence. Similarly, the terms "one", "another", and similar terms are not limited to one or more, but can include one or more. The terms "connected", "coupled", or similar terms are not limited to a direct connection or coupling, but can include an indirect connection or coupling, unless otherwise defined. The terms "upper", "lower", "left", "right", and similar terms are used herein merely to indicate relative positional relationships, and can change when the absolute positions of the described objects change.

[0081] As shown in Figure 1 The embodiment of the present application provides a multi-scale visual positioning method based on semantic consistency guidance, which comprises the following steps:

[0082] The text description corresponding to the input image is obtained, the text content is formatted, and then the input text description is analyzed by using a Stanza syntax analysis model to identify the syntax boundary and composition structure of the noun phrase in the sentence, and the text content corresponding to the noun phrase is extracted;

[0083] The input image, the text description, and the noun phrase are encoded in multiple modes based on a pre-trained BEiT-3 image-text encoding model, the input image generates visual features, the text description and the noun phrase generate overall text semantic features and noun phrase semantic features, respectively;

[0084] The text consistency constraint loss is calculated through the attention interaction between the noun phrase features and the visual features, and when the cross-attention calculation is performed, the attention weight and the response distribution of the noun phrase on each visual region are calculated, and a concept-level semantic heat map is generated;

[0085] The visual features are generated into multi-scale visual features through a feature pyramid network, and each scale of visual features is weighted according to the response distribution in the semantic heat map, the spatial intensity of the heat map is mapped into a weighting factor and acts on the feature maps of different scales, and the spatial position and channel response of the multi-scale features are executed with semantic-guided weighted processing;

[0086] A candidate region generation mechanism is constructed based on the multi-scale visual features and the semantic heat map, the semantic related regions are enhanced through the weighted features, and the objectness prediction and the bounding box regression are completed by the candidate region generation head, so as to obtain a multi-scale candidate region set highly related to the text description;

[0087] The visual features of the query image and the gallery image in the test set are extracted using the trained candidate box generation model, and cross-modal similarity calculation is performed on the finally generated candidate region box in combination with the cross-modal matching model; the target region box most relevant to the semantic description is output by sorting the candidate region boxes according to the similarity, and the positioning task is completed.

[0088] In the above embodiment, the multi-scale visual positioning method based on semantic consistency guidance can be mainly divided into a training stage and an application stage, as shown in Figure 2 The specific steps of each stage are as follows:

[0089] Training stage:

[0090] S1: input the input image corresponding to the text description into the Stanza syntactic analysis model to perform syntactic dependency analysis on the text description to extract the noun phrases in the sentence and the related syntax, and obtain the noun phrases .

[0091] The image and the text description come from the training set of CIR and LaSCo datasets; CIR dataset is a cross-image retrieval and visual reasoning benchmark, which is composed of 36,018 real scene images, including about 7,000 training triplets, 1,000 validation triplets and 2,000 test triplets; each sample is composed of a reference image, an artificially written difference description and a target image; LaSCo dataset is a large-scale combined reference segmentation benchmark, which includes 121,479 natural images, more than 169,000 reference representations and corresponding pixel-level segmentation masks, and is divided into training, validation and test sets; each reference representation covers complex descriptions such as multi-attribute combination, spatial relationship and multi-target interaction.

[0092] S2: input the input image , the text description and the noun phrase into the image-text encoding model to extract the visual features , the text semantic features and the phrase semantic features .

[0093] ​​​S3: calculating the text consistency constraint loss through the attention interaction between the noun phrase features and the visual features, calculating the attention weight of the noun phrase on each visual region and the response distribution when performing cross-attention calculation, and generating a concept-level semantic heat map;

[0094] S3.1: constructing a context version text segment by embedding each noun phrase into its corresponding position in the original sentence while preserving its surrounding dependency context, and inputting the visual feature set and the context version text segment into the cross-modal attention module to obtain a context attention map, the calculation process of which satisfies the following relationship:

[0095] , , ;

[0096] ;

[0097] In the formula, , , respectively represent the query, key, and value of the noun phrase under the context version text segment condition, represents the i-th noun phrase semantic feature, represents the context version text segment corresponding to the i-th noun phrase, represents the cross-modal attention map of the i-th noun phrase under the context version text segment condition, , , respectively represent three fully connected layers of the noun phrase under the context version text segment condition, represents an activation function, represents the dimension of the text feature;

[0098] S3.2: inputting the visual feature set and the noun phrase into the cross-modal attention module to obtain an independent phrase attention map, the calculation process of which satisfies the following relationship:

[0099] , , ;

[0100] ;

[0101] In the formula, , , respectively represent the query, key, and value when the i-th noun phrase is independently input, represents the cross-modal attention map when the i-th noun phrase is independently input, , , respectively represent three fully connected layers of the noun phrase under the independent condition;

[0102] S3.3: The context attention map and the independent phrase attention map are differentially processed and a consistency difference matrix is generated through a non-negative operation. Then, all the consistency difference matrices are accumulated according to the noun phrase index to form a text consistency constraint loss, and the calculation process satisfies the following relationship:

[0103] ;

[0104] ;

[0105] In the formula, differs from the difference matrix of the i-th noun phrase, represents an activation function, represents a threshold value, represents a text consistency loss function;

[0106] S3.4: Finally, a concept-level semantic heat map is generated based on the spatial response of the context attention map, and the generation process satisfies the following relationship:

[0107] ;

[0108] In the formula, represents a heat map, represents an attention map merging function.

[0109] S4: The feature pyramid network first performs down-sampling processing on the visual features at different scales, and then applies convolution mapping to the down-sampled features to obtain a multi-level visual feature set, and the sampling process satisfies the following relationship:

[0110] ;

[0111] , ;

[0112] In the formula, represents a multi-level visual feature map set, represents a visual feature map at the s-th scale, represents a construction convolution function at the s-th scale, represents down-sampling of visual features at different scales, represents the number of levels of the multi-scale visual feature map.

[0113] S5: According to the response distribution in the semantic heat map, corresponding weights are imposed on each scale visual feature, the spatial intensity of the heat map is mapped into a weighted factor and acts on the feature map of different scales, and the spatial position and channel response of the multi-scale feature are subjected to semantic-oriented weighted processing;

[0114] S5.1: The semantic heat map is subjected to scale alignment according to the spatial size of each scale feature map, and a heat map representation consistent with the corresponding scale is obtained, and the alignment process satisfies the following relationship:

[0115] ;

[0116] In the formula, represents the aligned heat map of the s-th scale visual feature, represents a scale transformation function, represents a spatial scale;

[0117] S5.2: The aligned heat map is subjected to convolution mapping and nonlinear transformation to generate a weight matrix for adjusting the intensity of the visual feature, and the generation process satisfies the following relationship:

[0118] ;

[0119] In the formula, represents a 1x1 scale convolution function, represents a hyperparameter, represents an activation function, represents the weighting matrix of the s-th scale;

[0120] S5.3: Finally, the weight matrix and the corresponding scale visual feature are subjected to point-by-point multiplication, thereby imposing differentiated semantic weights on the visual response of different spatial positions, realizing cross-scale semantic-guided feature enhancement, and the weighted scale visual feature is:

[0121] ;

[0122] In the formula, represents the weighted s-th scale visual feature, represents element-wise multiplication.

[0123] S6: Based on the multi-scale visual feature and the semantic heat map, a candidate region generation mechanism is constructed, the semantic related region is enhanced by the weighted feature, and the objectness prediction and the bounding box regression are completed by the candidate region generation head;

[0124] S6.1: In each scale s, the weighted visual feature is input into the objectness prediction module, and after the object prediction value is converted into the objectness confidence, the calculation process satisfies the following relationship:

[0125] ;

[0126] ;

[0127] In the formula, and These represent the feature maps respectively. Axis coordinates and Axis coordinates Represents the s-th scale position Predicted object properties at location, This represents the classification convolution function at the s-th scale. Indicates the weighted s-th scale position Visual features of the location Represents the s-th scale position Object confidence level at the location;

[0128] S6.2: Introduce a boundary regression module to predict the offsets of candidate boxes in four directions. The offset regression process satisfies the following relationship:

[0129] ;

[0130] In the formula Represents the s-th scale position The boundary offset vector at that location, Let represent the regression convolution function at the s-th scale. , , , Representing the s-th scale position respectively Distance values ​​to the left, top, right, and bottom boundaries of the candidate region;

[0131] S6.3: Map the grid positions on the feature map back to the original image coordinates to generate candidate bounding box center points with actual physical meaning. The mapping relationship satisfies the following formula:

[0132] ;

[0133] In the formula, This represents the downsampling step size at the s-th scale. Represents the s-th scale position Corresponding to the center coordinates of the original image;

[0134] S6.4: Using geometric relationships, the coordinates of the top-left and bottom-right corners of the candidate box in the original image can be constructed, thus forming a standard rectangular bounding box representation. The construction process satisfies the following relationship:

[0135] ;

[0136] ;

[0137] ;

[0138] ;

[0139] wherein, and represent the x-axis coordinate and y-axis coordinate of the top-left corner of the candidate box, respectively, and represent the x-axis coordinate and y-axis coordinate of the bottom-right corner of the candidate box, respectively;

[0140] S6.5: For each candidate position, construct a comprehensive confidence score for it to reflect the information of both the dimensions of “whether the region is an object” and “whether the region is relevant to the text semantics”, and then perform a threshold filtering on the entire candidate box set according to the comprehensive score, and the calculation and filtering process satisfy the following relationship:

[0141] ;

[0142] ;

[0143] wherein, denotes the comprehensive score at the s-th scale position , denotes a hyperparameter, denotes the heat map after visual feature alignment at the s-th scale position , denotes the filtered candidate box set at the s-th scale position , denotes a score threshold.

[0144] The application uses a BEiT-3-based image-text feature extraction network to perform multi-scale feature coding with a resolution of 384x384 on an input image, and performs standardization and noun phrase extraction processing on the text description; random flipping, color disturbance, random erasing and other enhancement strategies are used on the image side to improve the diversity of training samples. The training process is divided into two stages: stage 1: semantic consistency association optimization. In this stage, 32 epochs are trained, the cross-modal attention interaction between the semantic features of the noun phrases and the visual features is used to construct the text consistency constraint loss, and the concept-level semantic heat map is generated according to the attention response to optimize the semantic alignment capability of the model; stage 2: multi-scale visual target positioning optimization. The semantic heat map generated in stage 1 is used as a priori to guide the multi-scale visual features output by the feature pyramid network, and adaptive sampling and multi-scale candidate box generation are performed through the deformable attention module to improve the positioning accuracy and region pointing capability. In all training stages, the AdamW optimizer is used, and the cosine learning rate scheduling strategy is adopted, with an initial learning rate of 2e-5 and a batch size of 32. All methods are implemented on a single NVIDIA A800 80GB GPU using PyTorch, and the pytorch framework is used;

[0145] Application stage:

[0146] S7: using the trained candidate box generation model to directly generate candidate region boxes and their corresponding visual features for test set images; then performing cross-modal similarity calculation on the candidate box visual features and the semantic features of the text description; sorting all candidate boxes according to the similarity size, and selecting the candidate box closest to the text semantics as the final output to complete the positioning task.

[0147] The disclosure embodiment also provides a multi-scale visual positioning system based on semantic consistency guidance, comprising a processor, a memory;

[0148] The memory is used to store computer programs;

[0149] The processor is used to execute the programs stored on the memory, and realizes any method in the multi-scale visual positioning method based on semantic consistency guidance.

[0150] The above detailed the preferred embodiments of the application. It should be understood that those skilled in the art can make many modifications and changes without creative labor according to the concept of the application. Therefore, any technical solution obtained by logical analysis, reasoning or limited experiment based on the existing technology according to the concept of the application shall be within the protection scope determined by the claims.

Claims

1. A multi-scale visual localization method based on semantic consistency guidance, characterized in that, include: The text description corresponding to the input image is obtained. After formatting the text content, the Stanza syntactic analysis model is used to perform syntactic dependency analysis on the input text description, identify the grammatical boundaries and compositional structure of noun phrases in the sentence, and extract the text content corresponding to the noun phrases. Based on the pre-trained BEiT-3 image-text coding model, multimodal coding is performed on the input image, text description, and noun phrases. The input image generates visual features, and the text description and noun phrases generate overall text semantic features and noun phrase semantic features, respectively. The text consistency constraint loss is calculated by the attentional interaction between semantic features and visual features of noun phrases. When performing cross-attention calculation, the attention weights and response distributions of noun phrases in each visual region are calculated, and a concept-level semantic heatmap is generated. Visual features are generated through a feature pyramid network to generate multi-scale visual features. Based on the response distribution in the semantic heatmap, corresponding weights are applied to visual features at each scale. The spatial intensity of the heatmap is mapped to a weighting factor and applied to feature maps at different scales. Semantic-guided weighting is performed on the spatial location and channel response of the multi-scale features. A candidate region generation mechanism is constructed based on multi-scale visual features and semantic heatmaps. The semantically relevant regions are enhanced by weighted features, and the candidate region generation head completes object prediction and bounding box regression, thereby obtaining a set of multi-scale candidate regions that are highly related to the text description. The trained candidate bounding box generation model is used to directly generate candidate region bounding boxes and their corresponding visual features on the test set images. Then, cross-modal similarity calculation is performed between the visual features of these candidate boxes and the semantic features of the text description; all candidate boxes are sorted according to the similarity, and the candidate box that is closest to the semantics of the text is selected as the final output to complete the localization task.

2. The multi-scale visual localization method based on semantic consistency guidance according to claim 1, characterized in that, The process involves obtaining the text description corresponding to the input image, performing preprocessing to standardize the text format, and then performing syntactic dependency analysis on the preprocessed text description based on the Stanza syntactic analysis model. This identifies the grammatical boundaries and structural composition of noun phrases within the sentence and extracts the text content corresponding to the noun phrases, including: Input image Corresponding text description Preprocessing is performed to standardize the text format, and the preprocessed text is then... The text is input into the Stanza syntactic analysis model, where syntactic dependency parsing is performed to extract noun phrases and related grammatical relations. ; Among them, the input image and text description All data are from the training sets of the CIR and LaSCo datasets.

3. The multi-scale visual localization method based on semantic consistency guidance according to claim 1, characterized in that, Multimodal encoding is performed on the input image, text description, and noun phrases. Visual features are generated from the input image, and overall text semantic features and noun phrase semantic features are generated from the text description and noun phrases, respectively. These features include: Input image Text description and noun phrases Input into the image-text coding model and extract respectively visual features , Textual semantic features and Phrasal semantic features .

4. The multi-scale visual localization method based on semantic consistency guidance according to claim 1, characterized in that, The text consistency constraint loss is calculated through the attentional interaction between semantic and visual features of noun phrases. During cross-attention calculation, the attention weights and response distributions of noun phrases in each visual region are calculated, and a concept-level semantic heatmap is generated, including: By embedding each noun phrase into its corresponding position in the original sentence and preserving its surrounding dependency context, a contextual version of the text fragment is constructed. The visual feature set and the contextual version of the text fragment are then input into a cross-modal attention module to obtain a contextual attention map. The calculation process satisfies the following relationship: , , ； ； In the formula, , , These represent the query, key, and value of a noun phrase within the context of a text fragment. Representing the visual features of an image, This represents the semantic features of the i-th noun phrase. This represents the contextual version of the text fragment corresponding to the i-th noun phrase. This represents the cross-modal attention graph of the i-th noun phrase under the condition of the context version of the text segment. , , These represent three fully connected layers representing noun phrases within the context of different text fragments. This represents the activation function. Dimensions representing text features; The visual feature set and noun phrases are input into the cross-modal attention module to obtain an independent phrase attention map. The calculation process satisfies the following relationship: , , ； ； In the formula, , , Let represent the query, key, and value when the i-th noun phrase is entered independently. This represents the cross-modal attention graph when the i-th noun phrase is input independently. , , These represent three fully connected layers for noun phrases under independent conditions; The contextual attention map and the independent phrase attention map are differencing and then nonnegated to generate a consistency difference matrix. Subsequently, all consistency difference matrices are summed according to the noun phrase index to form the text consistency constraint loss. The calculation process satisfies the following relationship: ； ； In the formula, This represents the difference matrix of the i-th noun phrase. This represents the activation function. Indicates the threshold. Represents the text consistency loss function; Finally, a concept-level semantic heatmap is generated based on the spatial response of the context attention map. The generation process satisfies the following relationship: ； In the formula, Represents a heat map. This indicates the function for merging attention graphs.

5. The multi-scale visual localization method based on semantic consistency guidance according to claim 1, characterized in that, Visual features are generated through a feature pyramid network to create multi-scale visual features, including: Feature pyramid network first analyzes visual features Different downsampling rates are applied, and then convolution mapping is applied to the downsampling features to obtain a multi-level visual feature set. The sampling process satisfies the following relationship: ； ， ； In the formula, Represents a set of multi-level visual feature maps. This represents the visual feature map at the s-th scale. This represents the construction of the convolution function at the s-th scale. This indicates that visual features are downsampled at different scaling ratios. This represents the number of levels in a multi-scale visual feature map.

6. The multi-scale visual localization method based on semantic consistency guidance according to claim 1, characterized in that, Based on the response distribution in the semantic heatmap, corresponding weights are applied to visual features at each scale. The spatial intensity of the heatmap is mapped into weighting factors and applied to feature maps at different scales. Semantic-guided weighting processing is performed on the spatial location and channel response of multi-scale features, including: The semantic heatmap is scale-aligned according to the spatial dimensions of the feature maps at each scale to obtain a heatmap representation consistent with the corresponding scale. The alignment process satisfies the following relationship: ； In the formula, Represents a heat map. This represents the heatmap after aligning the visual features at the s-th scale. This represents the visual feature map at the s-th scale. Represents the scaling function. Indicates spatial scale; The aligned heatmap is subjected to convolutional mapping and nonlinear transformation to generate a weight matrix for adjusting the intensity of visual features. The generation process satisfies the following relationship: ； In the formula, This represents a 1×1 scale convolution function. Indicates hyperparameters, This represents the activation function. Let represent the weighting matrix at the s-th scale; Finally, the weight matrix is ​​multiplied pointwise with the visual features at the corresponding scales to apply differentiated semantic weights to the visual responses at different spatial locations, thereby achieving cross-scale semantic guidance feature enhancement. The weighted scale visual features are as follows: ； In the formula, This represents the weighted visual feature at the s-th scale. This indicates element-wise multiplication.

7. The multi-scale visual localization method based on semantic consistency guidance according to claim 1, characterized in that, A candidate region generation mechanism is constructed based on multi-scale visual features and semantic heatmaps. Weighted features are used to enhance semantically relevant regions, and the candidate region generation head completes object prediction and bounding box regression, including: At each scale s, weighted visual features will be used. The data is input into the object prediction module, and after obtaining the object prediction value, it is converted into an object confidence score. The calculation process satisfies the following relationship: ； ； In the formula, and These represent the feature maps respectively. Axis coordinates and Axis coordinates Represents the s-th scale position Predicted object properties at location This represents the classification convolution function at the s-th scale. Indicates the weighted s-th scale position Visual features of the location Represents the s-th scale position Object confidence level at the location; A boundary regression module is introduced to predict the offsets of candidate boxes in four directions. The offset regression process satisfies the following relationship: ； In the formula Represents the s-th scale position The boundary offset vector at that location, This represents the regression convolution function at the s-th scale. , , , Representing the s-th scale position respectively Distance values ​​to the left, top, right, and bottom boundaries of the candidate region; The grid positions on the feature map are mapped back to the original image coordinates to generate candidate bounding box center points with actual physical meaning. The mapping relationship satisfies the following formula: ； In the formula, This represents the downsampling step size at the s-th scale. Represents the s-th scale position Corresponding to the center coordinates of the original image; Using geometric relationships, the coordinates of the top-left and bottom-right corners of the candidate bounding box in the original image can be constructed, thus forming a standard rectangular bounding box representation. The construction process satisfies the following relationship: ； ； ； ； In the formula, and These represent the x-coordinate and y-coordinate of the top-left corner of the candidate box, respectively. and These represent the x-axis and y-axis coordinates of the bottom right corner of the candidate box, respectively. For each candidate location, a comprehensive confidence score is constructed to simultaneously reflect information from two dimensions: "whether the region is an object" and "whether the region is semantically relevant to the text." Then, the entire candidate box set is subjected to a threshold filter based on the comprehensive score. The calculation and filtering process satisfy the following relationship: ； ； In the formula, Represents the s-th scale position The overall score of the location Indicates hyperparameters, Represents the s-th scale position Heatmap of visual features aligned at the location. Represents the s-th scale position The filtered candidate box set, This indicates the score threshold.

8. The multi-scale visual localization method based on semantic consistency guidance according to claim 1, characterized in that, The trained candidate bounding box generation model is used to directly generate candidate region bounding boxes and their corresponding visual features on the test set images. Then, cross-modal similarity calculation is performed between the visual features of these candidate boxes and the semantic features of the text description; all candidate boxes are sorted according to the similarity, and the candidate box that is closest to the semantics of the text is selected as the final output to complete the localization task.

9. A multi-scale visual localization system guided by semantic consistency, characterized in that, Including processor and memory; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method described in any one of claims 1-8.

Images