Visual positioning method and device, electronic equipment and storage medium
By obtaining the text and images entered by the user, the fusion vector processing and image segmentation are performed, and layer -by -layer processing is performed on each feature image, the problem of long vision positioning time in existing technology is solved, and faster items processing is achieved. And mark.
Patent Information
- Application Number
- CN202411414586.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-10
- Publication Date
- 2025-01-28
AI Technical Summary
In the prior art, the visual positioning processing time is long and it is impossible to effectively process newly emerging items, so data collection and retraining are required.
By obtaining the text and images entered by the user, the fusion vector is processed, the segmentation image is a multi -scale feature diagram, and the image of the target positioning is marked with each feature sub -image.
The processing time of visual positioning has been increased, and new items can be processed and labeled more quickly, reducing the needs of data collection and re -training.
Smart Images
Figure CN119379975A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision technology, and in particular to a visual positioning method, device, electronic equipment and storage medium. Background Art
[0002] The open world explores and transforms a dynamic and simulated world through the subjective and proactive application of a large number of coupling rules. With the widespread application of the open world in social, entertainment and other fields, virtual directors and digital cultural relics often use corresponding visual positioning algorithms for visual positioning.
[0003] Current visual positioning algorithms, such as image segmentation and object detection, require data collection and retraining for newly appeared objects, and then use the trained data to visually position the trained objects. Since the above method can only process trained objects, data collection and retraining are required for new objects, resulting in a long processing time for visual positioning. Summary of the invention
[0004] In view of this, embodiments of the present invention provide a visual positioning method, device, electronic device and storage medium to solve the problem of long processing time of visual positioning in the prior art.
[0005] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:
[0006] A first aspect of an embodiment of the present invention shows a visual positioning method, the method comprising:
[0007] Get the text entered by the user and the image of the selected open world;
[0008] Performing processing based on the text and the image to obtain a fusion vector;
[0009] Segmenting the image to obtain a multi-scale feature map, wherein the multi-scale feature map includes a plurality of feature sub-images;
[0010] For each characteristic sub-image, image processing is performed layer by layer based on the fusion vector and the characteristic sub-image to obtain an image marked with target location, and output it.
[0011] Optionally, processing is performed based on the text and the image to obtain a fusion vector, including:
[0012] Performing vectorization processing on the text to obtain a text vector;
[0013] Processing the image using a visual model based on an attention mechanism to obtain an image global vector;
[0014] The text vector and the image global vector are connected to obtain a fusion vector.
[0015] Optionally, segmenting the image to obtain a multi-scale feature map includes:
[0016] Extracting features from the image to obtain a first feature map, wherein the first feature map has a plurality of layers;
[0017] Upsampling the first feature map to obtain a second feature map, where the number of layers of the second feature map is the same as the number of layers of the first feature map;
[0018] For each number of layers, the sub-feature map of the corresponding layer in the first feature map is concatenated with the sub-feature map of the corresponding layer in the second feature map to obtain a multi-scale feature map, where the multi-scale feature map includes multiple feature sub-images.
[0019] Optionally, for each characteristic sub-image, performing image layer-by-layer processing based on the fusion vector and the characteristic sub-image to obtain an image marked with target location includes:
[0020] For each feature sub-image in the multi-scale feature map, the feature sub-image and the fusion vector are input into the corresponding feature attention layer for processing to obtain corresponding feature fusion data;
[0021] If it is determined that the current feature sub-image is the feature sub-image numbered 2 in the multi-scale feature map, the feature fusion data output by the previous feature attention layer and the feature fusion data corresponding to the current feature sub-image are input to the upward connection layer for processing to obtain connection data; if it is determined that the current feature sub-image is the feature sub-image numbered 1 in the multi-scale feature map, no processing is performed;
[0022] If it is determined that the current feature sub-image is a feature sub-image with a number greater than 2, the connection data output by the upward connection layer corresponding to the previous feature sub-image and the feature fusion data corresponding to the current feature sub-image are input into the next upward connection layer for processing to obtain connection data;
[0023] If it is determined that the multi-scale feature map has been processed, the connection data output by the upward connection layer is expanded upward, and an image marked with the target location is output.
[0024] Optionally, for each feature sub-image in the multi-scale feature map, the feature sub-image and the fusion vector are input into a corresponding feature attention layer for processing to obtain corresponding feature fusion data, including:
[0025] For each feature sub-image in the multi-scale feature map, the fusion vector is processed based on the self-attention mechanism in the feature attention layer FAB to obtain a self-attention mechanism vector;
[0026] Processing the self-attention mechanism vector based on the fully connected layer and the activation function in the feature attention layer to obtain an attention feature;
[0027] The attention feature and the image feature of the feature sub-image are multiplied to obtain feature fusion data.
[0028] Optionally, the feature fusion data output by the previous feature attention layer and the feature fusion data corresponding to the current feature sub-image are input to the upward connection layer for processing to obtain connection data, including:
[0029] The feature fusion data output by the previous feature attention layer is connected with the feature fusion data corresponding to the current feature sub-image by using the upward connection layer to obtain the connected feature fusion data;
[0030] The upward connection layer is used to perform multi-layer convolution on the connection feature fusion data to output connection data.
[0031] Optionally, the connection data output by the upward connection layer is upwardly expanded to output an image marked with the target location, including:
[0032] Use the upward expansion layer to perform multi-layer convolution on the connection data output by the upward connection layer to obtain convolution data;
[0033] The convolution data is input into the activation function layer to obtain an image marked with the target location.
[0034] A second aspect of an embodiment of the present invention shows a visual positioning device, the device comprising:
[0035] An acquisition unit, used for acquiring text input by a user and an image of the selected open world;
[0036] A first processing unit, configured to perform processing based on the text and the image to obtain a fusion vector;
[0037] A second processing unit is used to perform segmentation processing on the image to obtain a multi-scale feature map, where the multi-scale feature map includes a plurality of feature sub-images;
[0038] The positioning unit is used to process each characteristic sub-image layer by layer based on the fusion vector and the characteristic sub-image, obtain an image marked with the target positioning, and output it.
[0039] A third aspect of an embodiment of the present invention shows an electronic device, which is used to run a program, wherein the program executes the visual positioning method shown in the first aspect of the embodiment of the present invention when running.
[0040] A fourth aspect of an embodiment of the present invention shows a storage medium, which includes a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute the visual positioning method shown in the first aspect of the embodiment of the present invention.
[0041] Based on the above-mentioned embodiment of the present invention, a visual positioning method, device, electronic device and storage medium are provided, the method comprising: obtaining text input by a user and an image in a selected open world; processing based on the text and the image to obtain a fusion vector; segmenting the image to obtain a multi-scale feature map, the multi-scale feature map including multiple feature sub-images; for each feature sub-image, performing layer-by-layer image processing based on the fusion vector and the feature sub-image to obtain an image marked with target positioning, and output it. In the embodiment of the present invention, the input text is determined, the text and the image are first fused; the image is then segmented to obtain a multi-scale feature map composed of multiple feature sub-images; for each feature sub-image of the multi-scale feature map, the fusion vector and the feature sub-image are processed layer by layer, so that the object area corresponding to the text can be marked on the corresponding image, thereby labeling detection can be performed by asking questions to visually locate the image; thereby improving the processing time of visual positioning. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.
[0043] Figure 1 A schematic diagram of a flow chart of a visual positioning method according to an embodiment of the present invention;
[0044] Figure 2 A schematic diagram of a process of an image labeled with target positioning according to an embodiment of the present invention;
[0045] Figure 3 A schematic diagram of the architecture of a feature attention layer shown in an embodiment of the present invention;
[0046] Figure 4 A schematic diagram of the architecture of the upward connection layer shown in an embodiment of the present invention;
[0047] Figure 5 This is a schematic diagram of the architecture of the upward expansion layer shown in an embodiment of the present invention;
[0048] Figure 6 This is an example diagram of visual positioning shown in an embodiment of the present invention;
[0049] Figure 7 The figure is a schematic diagram of the architecture of a visual positioning device according to an embodiment of the present invention. DETAILED DESCRIPTION
[0050] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0051] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0052] It should be noted that the descriptions of "first", "second", etc. in the present invention are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.
[0053] In this application, the terms "comprises", "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element.
[0054] See also Figure 1, is a flow chart of a visual positioning method shown in an embodiment of the present invention, the method comprising:
[0055] Step S101: Acquire the text input by the user and the image of the selected open world.
[0056] In the specific implementation of step S101, the user may perform voice input through an input terminal, or directly input text. If it is voice input, the language is converted into text according to the semantic recognition device.
[0057] Next, select the open world image you want to use from the drop-down menu, such as an open world image of a virtual director, and use it as the image.
[0058] It should be noted that the text may be a prompt and the voice may be a description of the problem.
[0059] Step S102: Processing is performed based on the text and the image to obtain a fusion vector.
[0060] It should be noted that the specific process of implementing step S102 includes the following steps:
[0061] Step S11: vectorize the text to obtain a text vector.
[0062] In the specific implementation process of step S11, the content of the text is vectorized and encoded by a text encoder to obtain a text vector.
[0063] The text vector consists of the batch size, text length, and feature dimensions.
[0064] For example, if the text vector is [B, 48, 1024], B is the batch size, 48 is the text length, and 512 is the feature dimension.
[0065] Step S12: Process the image based on the visual model of the attention mechanism to obtain a global image vector.
[0066] In the process of implementing step S12, first, the size of the input image is converted into a standard size, and the standard size is input into a visual model of the attention mechanism, such as a visual model (Vision Transformer, Vit) network, so that the visual model directly processes the image block sequence to perform image classification tasks and target detection, etc., to obtain the image global vector of the image.
[0067] For example, convert the size of image a to 572*572*3. After feature extraction by the visual model, the dimension of the extracted features is [B,256+1,1024], where B is the batch size, 256 is the number of blocks, +1 is the dimension of the CLS placeholder, and 1024 is the dimension of the feature.
[0068] Step S13: Connect the text vector and the image global vector to obtain a fusion vector.
[0069] In an embodiment of the present invention, the text and the image are vectorized and then fused. The deep learning model Transformer architecture can be used to fuse the information of the text and the image, so as to subsequently comprehensively predict the question, that is, the location of the image corresponding to the text.
[0070] Step S103: performing segmentation processing on the image to obtain a multi-scale feature map.
[0071] In step S103, the multi-scale feature map includes a plurality of feature sub-images;
[0072] It should be noted that the specific implementation of step S103 includes the following steps:
[0073] Step S21: extracting features from the image to obtain a first feature map, wherein the first feature map has a plurality of layers;
[0074] In the specific implementation of step S21, the feature channel under the encoder in the semantic segmentation model is used to extract features of the image to obtain a multi-level feature map constructed by multiple image features, that is, a first feature map.
[0075] It should be noted that each layer of the first feature map includes at least one feature.
[0076] The semantic segmentation model can be a segmentation module U-net network.
[0077] Step S22: upsampling the first feature map to obtain a second feature map, where the number of layers of the second feature map is the same as the number of layers of the first feature map;
[0078] In the specific implementation of step S22, the first feature map is upsampled using the decoder in the semantic segmentation model to obtain image features with the same number of layers as in the first feature map, and a hierarchical feature map, i.e., a second feature map, is constructed based on this.
[0079] Step S23: For each number of layers, concatenate the sub-feature map of the corresponding layer in the first feature map with the sub-feature map of the corresponding layer in the second feature map to obtain a multi-scale feature map.
[0080] In step S23, the multi-scale feature map includes a plurality of feature sub-images.
[0081] In the process of implementing step S23, the first feature map is spliced with the sub-feature map of the same layer in the second feature map through the jump connection in the semantic segmentation model to obtain the feature sub-image corresponding to each level, and then the feature sub-image corresponding to each level is combined to obtain a multi-scale feature map.
[0082] It should be noted that the multi-scale feature map includes multiple feature sub-images, and each feature sub-image is marked in order from 1 to M, that is, numbered, and M is a positive integer greater than or equal to 2.
[0083] Step S104: for each characteristic sub-image, perform image processing layer by layer based on the fusion vector and the characteristic sub-image to obtain an image marked with target location, and output it.
[0084] It should be noted that, in the specific implementation of step S104, Figure 2 As shown, the following steps are included:
[0085] Step S201: For each feature sub-image in the multi-scale feature map, the feature sub-image and the fusion vector are input into the corresponding feature attention layer for processing to obtain corresponding feature fusion data.
[0086] It should be noted that the specific implementation of step S201 includes the following steps:
[0087] Step S31: For each feature sub-image in the multi-scale feature map, the fusion vector is processed based on the self-attention mechanism in the feature attention layer FAB to obtain a self-attention mechanism vector.
[0088] In the specific implementation process of step S31, for each feature sub-image in the multi-scale feature map, the fusion vector is input into the 3-layer self-attention mechanism in the feature attention layer FAB to compare the text vector in the fusion vector with each feature vector under the image global vector, that is, the text vector and the image global vector are fused to capture the global and local connections corresponding to the text and the image, and obtain the self-attention mechanism vector.
[0089] Step S32: Processing the self-attention mechanism vector based on the fully connected layer and activation function in the feature attention layer to obtain an attention feature;
[0090] In the specific implementation process of step S32, the self-attention mechanism vector is input into the fully connected layer MLP for classification training. After the classification training is completed, the high-dimensional vector cls embedding containing text information is obtained; then clsembedding is input into the activation function softmax for processing, and the corresponding attention feature is output.
[0091] Step S33: multiply the attention feature and the image feature of the feature sub-image to obtain feature fusion data.
[0092] In the process of implementing step S33, the feature sub-image is reshaped to obtain a new feature dimension, which is point-wise multiplied with the feature dimension of the attention feature to obtain feature fusion data; and the feature fusion data is reshaped to determine new feature fusion data.
[0093] It should be noted that each feature sub-image in the multi-scale feature map is processed through the above steps S31 to S33, so as to obtain feature fusion data corresponding to each feature sub-image.
[0094] In order to better understand the contents shown in the above steps S31 to S33, Figure 3 As shown, an example is given below to illustrate.
[0095] The input of the FAB layer is the feature sub-image and the fusion vector, where the dimension of the feature sub-image is [B, 280, 280, 128] and the dimension of the fusion vector is [B, 306, 1024].
[0096] The fusion vector first passes through a 3-layer self-attention mechanism to fuse the image embedding and text embedding to obtain a self-attention mechanism vector. At this time, the dimension corresponding to the self-attention mechanism vector is [B, 306, 1024].
[0097] The self-attention mechanism vector is processed by a fully connected layer MLP, and the feature dimension obtained is [B, 306, 128]. At this time, cls embedding is taken.
[0098] After the cls embedding passes through the softmax layer, the attention feature is obtained, and its dimension is [B, 1, 128]. After the image feature is reshaped, the feature dimension is [B, 280 2 ,128]; its [B,280 2 ,128] and the feature dimension [B,1,128] of the attention feature are multiplied point by point to obtain the feature fusion data [B,2802 ,128]; after reshaping, the dimension of the feature is [B,280,280,128].
[0099] It should be noted that after executing step S201, each characteristic sub-image needs to be processed in sequence.
[0100] Step 202: Determine whether the current feature sub-image is the feature sub-image numbered 2 in the multi-scale feature map. If it is the feature sub-image numbered 1, no processing is performed and the process returns to step S202. If it is the feature sub-image numbered 2, the process continues to step S203.
[0101] Since the image features need to be processed layer by layer, it is necessary to determine the visual positioning of the text in the image.
[0102] In the specific implementation of step S202, the current feature sub-image is obtained, and the position of the current feature sub-image in the multi-scale feature map is determined.
[0103] Step S203: Input the feature fusion data output by the previous feature attention layer and the feature fusion data corresponding to the current feature sub-image into the upward connection layer UC for processing to obtain connection data.
[0104] That is to say, if the current one is the second feature sub-image, the feature fusion data output by the first feature attention layer FAB and the feature fusion data output by the second feature attention layer FAB corresponding to the second feature sub-image are input into the upward connection layer to determine the corresponding connection data, and return to execute step S202 to continue monitoring the next feature sub-image.
[0105] It should be noted that N is a positive integer greater than or equal to 2, and the value of N is consistent with the number of feature sub-images in the multi-scale feature map.
[0106] It should be noted that the specific implementation of step S203 includes the following steps:
[0107] Step S41: Use the upward connection layer UC to connect the feature fusion data output by the previous feature attention layer and the feature fusion data corresponding to the current feature sub-image to obtain connected feature fusion data.
[0108] In the specific implementation process of step S41, the upward connection layer UC uses a bilinear interpolation method to up-sample the feature fusion data output by the previous feature attention layer, and connects the up-sampled data with the feature fusion data corresponding to the current feature sub-image to obtain connected feature fusion data.
[0109] Step S42: Use the upward connection layer UC to perform multi-layer convolution on the connection feature fusion data and output connection data.
[0110] In the specific implementation process of step S42, first, the connection feature fusion data is convolved using a convolution kernel, and then the convolved connection feature fusion data is processed by a linear rectification function to obtain multiple channel features; then, the convolution kernel is used to process it to obtain connection data.
[0111] In order to better understand the contents shown in the above steps S41 and S42, Figure 4 Give an example.
[0112] The input to the upward connection layer UC is the features [B, 64, 64, 512] output by the previous layer FAB and the features [B, 136, 136, 256] output by the current layer FAB.
[0113] The feature Imagefeature up of the upper layer will first be upsampled by the bilinear interpolation method to get the feature dimension of [B, 136, 136, 512], and it will be connected with the feature Imagefeature up now output by the FAB of this layer through the connection layer concate to get the feature dimension of [B, 136, 136, 768]; after convolution conv(stride=3x3, pad=1, input channal=768, output channal=768) and linear rectification function relu processing, the output feature is [B, 136, 136, 768]; convolution conv is performed again to get the feature dimension of [B, 280, 280, 896], and output is output.
[0114] Step 204: Determine whether the current feature sub-image is a feature sub-image with a number greater than 2. If so, execute step S205. If not, return to execute step S202.
[0115] Step S205: input the connection data outputted by the upward connection layer corresponding to the previous feature sub-image and the feature fusion data corresponding to the current feature sub-image into the next upward connection layer UC for processing to obtain connection data.
[0116] It should be noted that the specific implementation process of step S205 is the same as the specific implementation process of step S203 described above, and they can refer to each other.
[0117] Step S206: Determine whether the current feature sub-image is the Mth feature sub-image in the multi-scale feature map. If so, execute step S207; if not, execute step S202.
[0118] Wherein, M is the number of feature sub-images in the multi-scale feature map, and M is a positive integer greater than or equal to 2.
[0119] If it is determined that the current feature sub-image is not in the multi-scale feature map, or the current feature sub-image is empty, it means that the multi-scale feature map has been processed, that is, there is no processed feature sub-image, and step S207 is executed.
[0120] Step S207: Expand the connection data output by the upward connection layer UC upward, and output an image marked with the target location.
[0121] It should be noted that the specific process of implementing step S207 includes the following steps:
[0122] Step S51: Use the upward expansion layer US to perform multi-layer convolution on the connection data output by the upward connection layer UC to obtain convolution data.
[0123] In the specific implementation of step S51, the connection data is first convolved using a convolution kernel, and then the convolved connection data is processed by a linear rectification function to obtain multiple channel features; then the convolution kernel is used to process it to obtain convolution data.
[0124] Step S52: input the convolution data into the activation function layer, and output an image marked with the target location.
[0125] In the specific implementation of step S52, the convolution data is upsampled to obtain the features of the text in the image, which are then input into the activation function layer to determine the features with the highest probability, perform visual positioning, i.e., target positioning, and output an image marked with the target positioning.
[0126] It should be noted that the method of marking the target location in the output image marked with the target location may be text marking or area box marking.
[0127] It should be noted that visual positioning is a technology that determines the location of objects through visual perception and analysis of environmental information.
[0128] In order to better understand the contents shown in the above steps S51 and S52, Figure 5 Give an example.
[0129] The upward expansion layer US input is the connection data output by the upward connection layer UC, and the feature dimension of the connection data is [B, 280, 280, 896]. After a layer of convolution conv (stride = 3x3, pad = 1, input channal = 896, output channal = 256) and a linear rectification function relu layer, and then a layer of convolution conv (stride = 1x1, pad = 0, input channal = 256, output channal = 2), the feature [B, 280, 280, 2] is obtained, and the feature [B, 572, 572, 2] is obtained after upsampling using the bilinear difference method. After a layer of softmax layer, the prediction of text positioning grouding is obtained, that is, the target image marked with target positioning is obtained.
[0130] In order to better understand the process shown in the above steps S201 to S204, Figure 6 As shown, an example is given below to illustrate.
[0131] First, the image size is converted to 572*572*3. After feature extraction by Vit, the global image vector image embedding is obtained. Its feature dimension is [B, 256+1, 1024]. The text content text is input into the text encoder Text encoder for vectorization encoding to obtain the text vector Text embedding. Its feature dimension is [B, 48, 1024].
[0132] Text embedding and image global vector image embeding are input into the connection layer to connect to obtain the fused vector embedding, whose corresponding feature dimension is [B, 305, 1024]; a learnable clsembedding is added to the fused vector, and the feature dimension is [B, 306, 1024].
[0133] Next, the image image is input into the U-net network for segmentation processing to obtain a multi-scale feature map mutl-scaleimage feature, which includes 3 feature sub-images and is numbered, specifically including a feature sub-image imagefeature1 numbered 1, whose feature dimension is [B, 280, 280, 128], a feature sub-image imagefeature2 numbered 2, whose feature dimension is [B, 136, 136, 256], and a feature sub-image imagefeature3 numbered 3, whose feature dimension is [B, 64, 64, 512].
[0134] The feature dimension [B, 280, 280, 128] of image feature1 and the feature dimension [B, 306, 1024] of the fusion vector are input into the feature attention layer FAB corresponding to the feature sub-image image feature1 numbered 1. The fusion vector is fused with image embedding and text embedding through 3-layer self-attention layers. At this time, the feature dimension of the fusion vector is [B, 306, 1024].
[0135] After a fully connected layer MLP, the feature dimension is [B, 306, 128]. Take the cls embedding and get the attention feature after the softmax layer, and its dimension is [B, 1, 128]. The feature dimension after reshaping the image feature is [B, 280 2 ,128]; after multiplying it with the feature dimension of image feature1 [B,280,280,128], the output is [B,280 2 ,128], and then after reshaping, the feature fusion data is obtained, and its feature dimension is [B,280,280,128].
[0136] Similarly, the feature sub-image numbered 2, image feature2, and the feature sub-image numbered 3, image feature3, are both input into the corresponding FAB layer for the above output.
[0137] Then, the feature fusion data output by the feature attention layer FAB corresponding to the feature sub-image image feature1 numbered 1 and the feature fusion data output by the feature attention layer FAB corresponding to the feature sub-image image feature2 numbered 2 are input into the first upward connection layer UC for processing.
[0138] Specifically, the input to the upward connection layer UC layer includes the feature fusion data output by the previous FAB layer, whose feature dimension is [B, 64, 64, 512], and the feature fusion data output by the current FAB layer, whose feature dimension is [B, 136, 136, 256].
[0139] The feature fusion data output by the previous FAB layer will first be upsampled using the bilinear interpolation method to obtain a feature dimension of [B, 136, 136, 512]. The feature dimension [B, 136, 136, 256] output by the FAB layer of this layer is concatenated to obtain a feature dimension of [B, 136, 136, 768]. After two layers of conv (stride = 3x3, pad = 1, input channal = 768, output channal = 768) and relu layers, the concatenated data is output, and its corresponding feature dimension is [B, 136, 136, 768].
[0140] Similarly, the feature dimension output by the first upward connection layer UC and the feature fusion data output by the feature attention layer FAB corresponding to the feature sub-image imagefeature1 numbered 3 are input into the second upward connection layer UC for the above processing. They can refer to each other. The feature dimension corresponding to the connection data output by the last layer UC is [B, 280, 280, 896].
[0141] Finally, the connection data output by the second upward connection layer UC is input into the upward expansion layer US. Specifically, the upward expansion layer US passes the feature dimension [B, 280, 280, 896] corresponding to the connection data through a layer of conv (stride = 3x3, pad = 1, input channal = 896, output channal = 256) and relu layer, and then passes through a layer of conv (stride = 1x1, pad = 0, input channal = 256, output channal = 2) to obtain the feature [B, 280, 280, 2]. After upsampling using the bilinear difference method, the feature [B, 572, 572, 2] is obtained. After passing through a softmax layer, the prediction of the positioning grouding is obtained, that is, the image marked with the target location is obtained.
[0142] For example, the image image contains multiple pictures q of cultural relics, and the content of the text p is used to indicate a certain cultural relic a; the pictures q and the text p are processed as above. Figure 6 The output is to mark the region of cultural relics a in the image q, i.e., visual positioning.
[0143] For example, the image image is a picture w in a video shot by a virtual director, and the content of the text r is used to indicate a task y; the picture w and the text r are processed as above. Figure 6 The output is to mark the area of task y in image w, which is visual positioning.
[0144] In an embodiment of the present invention, an input text is determined, and the text and an image are first fused; the image is then segmented to obtain a multi-scale feature map composed of a plurality of feature sub-images; for each feature sub-image of the multi-scale feature map, the fusion vector and the feature sub-image are processed layer by layer, and the object area corresponding to the text can be marked on the corresponding image, thereby labeling detection can be performed by asking questions to visually locate the image; thereby improving the processing time of visual positioning.
[0145] Based on the method shown in the above embodiment of the present invention, the embodiment of the present invention also shows a visual positioning device, such as Figure 7 As shown, the device comprises:
[0146] An acquisition unit 701 is used to acquire text input by a user and an image in the selected open world;
[0147] A first processing unit 702, configured to perform processing based on the text and the image to obtain a fusion vector;
[0148] A second processing unit 703 is used to perform segmentation processing on the image to obtain a multi-scale feature map, where the multi-scale feature map includes multiple feature sub-images;
[0149] The positioning unit 704 is used to perform layer-by-layer image processing on each characteristic sub-image based on the fusion vector and the characteristic sub-image to obtain an image marked with the target location and output it.
[0150] The specific principles and execution processes of each unit in the visual positioning device disclosed in the above embodiment of the present invention are the same as the corresponding contents in the visual positioning method provided in the above embodiment of the present invention. Please refer to the corresponding parts of the visual positioning method disclosed in the above embodiment of the present invention, and will not be repeated here.
[0151] In an embodiment of the present invention, an input text is determined, and the text and an image are first fused; the image is then segmented to obtain a multi-scale feature map composed of a plurality of feature sub-images; for each feature sub-image of the multi-scale feature map, the fusion vector and the feature sub-image are processed layer by layer, and the object area corresponding to the text can be marked on the corresponding image, thereby labeling detection can be performed by asking questions to visually locate the image; thereby improving the processing time of visual positioning.
[0152] Optionally, based on the visual positioning device shown in the above embodiment of the present invention, the first processing unit 702 is specifically configured to:
[0153] Performing vectorization processing on the text to obtain a text vector;
[0154] Processing the image using a visual model based on an attention mechanism to obtain an image global vector;
[0155] The text vector and the image global vector are connected to obtain a fusion vector.
[0156] Optionally, based on the visual positioning device shown in the above embodiment of the present invention, the second processing unit 703 is specifically configured to:
[0157] Extracting features from the image to obtain a first feature map, wherein the first feature map has a plurality of layers;
[0158] Upsampling the first feature map to obtain a second feature map, where the number of layers of the second feature map is the same as the number of layers of the first feature map;
[0159] For each number of layers, the sub-feature map of the corresponding layer in the first feature map is concatenated with the sub-feature map of the corresponding layer in the second feature map to obtain a multi-scale feature map, where the multi-scale feature map includes multiple feature sub-images.
[0160] Optionally, based on the visual positioning device shown in the above embodiment of the present invention, the positioning unit 704 is specifically used for:
[0161] For each feature sub-image in the multi-scale feature map, the feature sub-image and the fusion vector are input into the corresponding feature attention layer for processing to obtain corresponding feature fusion data;
[0162] If it is determined that the current feature sub-image is the feature sub-image numbered 2 in the multi-scale feature map, the feature fusion data output by the previous feature attention layer and the feature fusion data corresponding to the current feature sub-image are input to the upward connection layer for processing to obtain connection data; if it is determined that the current feature sub-image is the feature sub-image numbered 1 in the multi-scale feature map, no processing is performed;
[0163] If it is determined whether the current feature sub-image is a feature sub-image with a number greater than 2, the connection data output by the upward connection layer corresponding to the previous feature sub-image and the feature fusion data corresponding to the current feature sub-image are input into the next upward connection layer for processing to obtain the connection data;
[0164] If it is determined that the multi-scale feature map has been processed, the connection data output by the upward connection layer is expanded upward, and an image marked with the target location is output.
[0165] Among them, for each feature sub-image in the multi-scale feature map, the feature sub-image and the fusion vector are input into the corresponding feature attention layer for processing to obtain corresponding feature fusion data, including:
[0166] For each feature sub-image in the multi-scale feature map, the fusion vector is processed based on the self-attention mechanism in the feature attention layer FAB to obtain a self-attention mechanism vector;
[0167] Processing the self-attention mechanism vector based on the fully connected layer and the activation function in the feature attention layer to obtain an attention feature;
[0168] The attention feature and the image feature of the feature sub-image are multiplied to obtain feature fusion data.
[0169] Among them, the feature fusion data output by the previous feature attention layer and the feature fusion data corresponding to the current feature sub-image are input to the upward connection layer for processing to obtain connection data, including:
[0170] The feature fusion data output by the previous feature attention layer is connected with the feature fusion data corresponding to the current feature sub-image by using the upward connection layer to obtain the connected feature fusion data;
[0171] The upward connection layer is used to perform multi-layer convolution on the connection feature fusion data to output connection data.
[0172] Among them, the connection data output by the upward connection layer is expanded upward, and the image with the target location is annotated, including:
[0173] Use the upward expansion layer to perform multi-layer convolution on the connection data output by the upward connection layer to obtain convolution data;
[0174] The convolution data is input into the activation function layer to obtain an image marked with the target location.
[0175] An embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory is used for program codes and data for open-world visual positioning processing, and the processor is used to call program instructions in the memory to execute steps shown in the open-world visual positioning method in the above embodiment.
[0176] An embodiment of the present application provides a storage medium, which includes a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute the open world visual positioning method shown in the above embodiment.
[0177] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.
[0178] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.
[0179] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A visual positioning method, characterized in that: The method comprises: Get the text entered by the user and the image of the selected open world; Processing is performed based on the text and the image to obtain a fusion vector; Segmenting the image to obtain a multi-scale feature map, wherein the multi-scale feature map includes a plurality of feature sub-images; For each characteristic sub-image, image processing is performed layer by layer based on the fusion vector and the characteristic sub-image to obtain an image marked with target location, and output it.
2. The method according to claim 1, characterized in that Processing is performed based on the text and the image to obtain a fusion vector, including: Performing vectorization processing on the text to obtain a text vector; Processing the image using a visual model based on an attention mechanism to obtain an image global vector; The text vector and the image global vector are connected to obtain a fusion vector.
3. The method according to claim 1, characterized in that The image is segmented to obtain a multi-scale feature map, including: Extracting features from the image to obtain a first feature map, wherein the first feature map has a plurality of layers; Upsampling the first feature map to obtain a second feature map, where the number of layers of the second feature map is the same as the number of layers of the first feature map; For each number of layers, the sub-feature map of the corresponding layer in the first feature map is concatenated with the sub-feature map of the corresponding layer in the second feature map to obtain a multi-scale feature map, where the multi-scale feature map includes multiple feature sub-images.
4. The method according to claim 1, characterized in that: For each characteristic sub-image, image processing is performed layer by layer based on the fusion vector and the characteristic sub-image to obtain an image marked with target location, including: For each feature sub-image in the multi-scale feature map, the feature sub-image and the fusion vector are input into the corresponding feature attention layer for processing to obtain corresponding feature fusion data; If it is determined that the current feature sub-image is the feature sub-image numbered 2 in the multi-scale feature map, the feature fusion data output by the previous feature attention layer and the feature fusion data corresponding to the current feature sub-image are input to the upward connection layer for processing to obtain connection data; if it is determined that the current feature sub-image is the feature sub-image numbered 1 in the multi-scale feature map, no processing is performed; If it is determined that the current feature sub-image is a feature sub-image with a number greater than 2, the connection data output by the upward connection layer corresponding to the previous feature sub-image and the feature fusion data corresponding to the current feature sub-image are input into the next upward connection layer for processing to obtain connection data; If it is determined that the multi-scale feature map has been processed, the connection data output by the upward connection layer is expanded upward, and an image marked with the target location is output.
5. The method according to claim 4, characterized in that For each feature sub-image in the multi-scale feature map, the feature sub-image and the fusion vector are input into the corresponding feature attention layer for processing to obtain corresponding feature fusion data, including: For each feature sub-image in the multi-scale feature map, the fusion vector is processed based on the self-attention mechanism in the feature attention layer to obtain a self-attention mechanism vector; Processing the self-attention mechanism vector based on the fully connected layer and the activation function in the feature attention layer to obtain an attention feature; The attention feature and the image feature of the feature sub-image are multiplied to obtain feature fusion data.
6. The method according to claim 4, characterized in that The feature fusion data output by the previous feature attention layer and the feature fusion data corresponding to the current feature sub-image are input to the upward connection layer for processing to obtain connection data, including: The feature fusion data output by the previous feature attention layer is connected with the feature fusion data corresponding to the current feature sub-image by using the upward connection layer to obtain the connected feature fusion data; The upward connection layer is used to perform multi-layer convolution on the connection feature fusion data to output connection data.
7. The method according to claim 4, characterized in that The connection data output by the upward connection layer is expanded upward to output an image marked with the target location, including: Use the upward expansion layer to perform multi-layer convolution on the connection data output by the upward connection layer to obtain convolution data; The convolution data is input into the activation function layer to obtain an image marked with the target location.
8. A visual positioning device, characterized in that: The device comprises: An acquisition unit, used for acquiring text input by a user and an image of the selected open world; A first processing unit, configured to perform processing based on the text and the image to obtain a fusion vector; A second processing unit is used to perform segmentation processing on the image to obtain a multi-scale feature map, where the multi-scale feature map includes a plurality of feature sub-images; The positioning unit is used to process each characteristic sub-image layer by layer based on the fusion vector and the characteristic sub-image, obtain an image marked with the target positioning, and output it.
9. An electronic device, characterized in that: The electronic device is used to run a program, wherein the program, when running, executes the visual positioning method as described in any one of claims 1-7.
10. A storage medium, characterized in that: The storage medium includes a storage program, wherein when the program is running, the device where the storage medium is located is controlled to execute the visual positioning method as described in any one of claims 1-7.