Image geographic positioning method, device and equipment based on multi-modal large model

Abstract

The invention provides an image geographic positioning method, device and equipment based on a multi-modal large model, and relates to the technical field of artificial intelligence, and the method comprises the steps: obtaining a training data set which comprises a plurality of images; for each image, performing global positioning system (GPS) coordinate characterization of equal earth projection on the GPS coordinate features of the image, and performing coding based on random Fourier features to obtain GPS coordinate feature vectors of the image; for each image, carrying out image-text collaborative coding on the image to obtain a collaborative coding feature vector generated by fusing the image feature and the text feature of the image; inputting the GPS coordinate feature vector and the collaborative coding feature vector into a large model, carrying out multi-modal comparison training on the large model based on the embedding similarity of the GPS coordinate feature vector and the collaborative coding feature vector to obtain a trained large model, inputting a to-be-positioned image into the trained large model, and carrying out multi-modal comparison training on the large model to obtain a to-be-positioned image. And outputting the longitude coordinate value and the latitude coordinate value of the to-be-positioned image.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an image geolocation method, device and equipment based on a multimodal large model. Background Art

[0002] With the popularity of global social media platforms, a large number of images that can be used for invention have accumulated in cyberspace. These images are not only huge in number, but also contain rich potential value. Geographical location information provides the geographical background or shooting location of the image, which is of great value for in-depth extraction of image information, such as cyberspace mapping, intelligence acquisition, image target positioning, social network analysis, public safety, urban planning, environmental monitoring, etc. However, according to statistical research and analysis, only about 2% of images on instant social platforms are marked with geographic locations, and only about 25% on related professional image websites, so image geolocation technology has a high practical application value.

[0003] Image geolocation refers to determining the geographical location of a given image or its content. It can serve downstream tasks such as image retrieval, image matching, and cross-modal image discovery. This technology is of great significance in various applications, such as navigation, tourism, reconnaissance, and security. Image geolocation technology can be divided into regional geolocation, natural and urban area positioning, and global geolocation according to the positioning range. Among them, global geolocation is the ultimate goal of image geolocation technology tasks. However, due to the huge number and diversity of global geographical landscape images, it is difficult to build a large image library covering the entire world. Therefore, traditional image retrieval methods based on image libraries are difficult to meet the requirements of timeliness and accuracy in solving global positioning solutions. The existing related technical methods divide the earth into discrete geographical units and transform the geolocation problem into a classification problem. However, such methods are limited by the limitations of predefined categories. When the geographical location of the image deviates from the center of the predefined category, it usually leads to large positioning deviations. Summary of the invention

[0004] 1. Technical issues to be resolved

[0005] In view of the above-mentioned shortcomings, the main purpose of the present invention is to provide an image geolocation method, device and equipment based on a multimodal large model. The large model is a novel multimodal collaborative alignment heuristic image positioning large model, which uses a pre-trained model to align the features of images, texts, and geographic coordinates in a high-dimensional common subspace, and enhances the mapping relationship between the image and its corresponding geographic location coordinates by constructing a multi-level embedded feature representation. Compared with existing image geolocation methods, this method has higher judgment accuracy and faster reasoning speed, and can be migrated to tasks other than image geolocation, and has good performance.

[0006] (II) Technical solution

[0007] In order to achieve the above-mentioned purpose, the first aspect of an embodiment of the present invention provides an image geolocation method based on a multimodal large model, including: obtaining a training data set, the training data set including multiple images; for each image, performing GPS coordinate representation of the GPS coordinate features of the image in iso-earth projection and encoding based on random Fourier features to obtain a GPS coordinate feature vector of the image; for each image, performing image-text collaborative encoding on the image to obtain a collaborative encoding feature vector generated by fusing the image features and text features of the image; inputting the GPS coordinate feature vector and the collaborative encoding feature vector into the large model, performing multimodal comparative training on the large model based on the embedding similarity of the GPS coordinate feature vector and the embedding similarity of the collaborative encoding feature to obtain a trained large model, inputting the image to be located into the trained large model, and outputting the longitude coordinate value and latitude coordinate value of the image to be located.

[0008] According to an embodiment of the present invention, the GPS coordinates of an image are represented by iso-earth projection GPS coordinates, including: standardizing the projection of the GPS coordinates of the image and minimizing the distortion inherent in the standard system to obtain the projection coordinates of the GPS coordinates; scaling the projection coordinates to obtain the scaled projection coordinates.

[0009] According to an embodiment of the present invention, the GPS coordinates of an image are encoded based on random Fourier features, including: constructing a coding layer, determining the frequency range required for coding the coding layer by presetting a fixed matrix, wherein each element in the fixed matrix obeys a Gaussian distribution; determining different coding frequencies in the random Fourier features based on the range of the element standard deviation; encoding different scale information of the projection coordinates based on different coding frequencies to obtain hierarchical features at different scales; transferring the hierarchical features through a feedforward neural network, wherein the activation function of each feedforward neural network processes the Fourier coded features separately and generates a corresponding result feature vector; and combining multiple result feature vectors according to row elements to obtain a GPS coordinate feature vector.

[0010] According to an embodiment of the present invention, an image is collaboratively encoded with text and image to obtain a collaboratively encoded feature vector generated by fusing the image features and text features of the image, including: extracting text titles and text content from the image to obtain text features; using a pre-trained text encoder to encode the text features into a text feature vector; using a pre-trained image encoding model to encode the image features of the image into an image feature vector; and using an attention mechanism to fuse the text feature vector and the image feature vector to obtain a collaboratively encoded feature vector.

[0011] According to an embodiment of the present invention, based on the embedding similarity of GPS coordinate feature vectors and co-encoded features, multimodal contrast training is performed on the large model, including: using contrastive learning to align the similarities between image features, text features and corresponding GPS coordinate features of an image, and minimizing the similarity between GPS coordinate features of other images.

[0012] According to an embodiment of the present invention, multimodal comparative training is performed on a large model based on the embedding similarity of GPS coordinate feature vectors and collaborative coding features, and also includes: dividing the training data set into multiple batches of training data; for each image in each batch of training data, constructing a set of randomly enhanced data sets, the randomly enhanced data sets including at least one image negative sample; for each image in each batch of training data, injecting random noise into the GPS coordinates of each image, the random noise being randomly sampled from a Gaussian distribution whose standard deviation is the standard deviation of the elements in a fixed matrix.

[0013] According to an embodiment of the present invention, the image to be located is input into a trained large model, and the longitude coordinate value and the latitude coordinate value of the image to be located are output, including: inputting the image to be located into a trained large model, and outputting the GPS coordinate feature vector corresponding to the image to be located; inverting the GPS coordinate feature vector to obtain the longitude coordinate value and the latitude coordinate value of the image to be located.

[0014] The second aspect of an embodiment of the present invention provides an image geo-positioning device based on a multimodal large model, including: an input-output module, used to input the image to be positioned into a trained large model, and output a GPS coordinate feature vector corresponding to the image to be positioned; an inversion module, used to invert the GPS coordinate feature vector to obtain the longitude coordinate value and latitude coordinate value of the image to be positioned; an inference module, used to input the image to be positioned into the trained large model, and output the longitude coordinate value and latitude coordinate value of the image to be positioned.

[0015] A third aspect of an embodiment of the present invention provides an electronic device, comprising: one or more processors; a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0016] The fourth aspect of the present invention further provides a computer-readable storage medium on which a computer program or instruction is stored, and when the computer program or instruction is executed by a processor, the steps of the above method are implemented.

[0017] (III) Beneficial effects

[0018] The technical solution of the embodiment of the present invention has at least the following beneficial effects:

[0019] (1) This method introduces multimodal co-alignment to directly optimize the similarity between image and text features and GPS coordinate features, reduce positioning errors, overcome the limitations of predefined categories, and improve positioning accuracy. In addition, through multimodal alignment and contrastive learning, the dependence on a large amount of training data is reduced, so that the model can achieve high positioning accuracy even with limited training data.

[0020] (2) By leveraging the comparative learning capabilities of the large model and constructing a multi-level embedded feature representation, the mapping relationship between the image and its corresponding geographic location coordinates is enhanced. This multimodal collaborative alignment technology improves the efficiency of image geospatial positioning.

[0021] (3) This method does not rely on predefined geographic categories, but instead achieves precise positioning on a global scale by directly comparing the similarity between image and text features and GPS coordinate features through a pre-trained large model.

[0022] (4) This method achieves efficient geolocation without the need for a large-scale image library by using a classification-based approach and contrastive learning to train a model to learn the complex relationships between images, texts, and geographic locations. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A flowchart of an image geolocation method based on a multimodal large model according to an embodiment of the present invention is schematically shown;

[0024] Figure 2 The following schematically shows an architecture diagram of a multimodal collaborative alignment model according to an embodiment of the present invention;

[0025] Figure 3 A structural block diagram of an image geo-positioning device based on a multi-modal large model according to an embodiment of the present invention is schematically shown;

[0026] Figure 4 A block diagram of an electronic device suitable for implementing the method described above according to an embodiment of the present invention is schematically shown. DETAILED DESCRIPTION

[0027] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0028] The method for multimodal collaborative training of a large model according to an embodiment of the present invention will be described in detail below with reference to the accompanying drawings.

[0029] Figure 1 The flowchart of the image geolocation method based on the multimodal large model according to an embodiment of the present invention is schematically shown. Figure 2The following schematically shows an architecture diagram of a multimodal collaborative alignment model according to an embodiment of the present invention.

[0030] like Figure 1 and Figure 2 As shown, the image geolocation method based on the multimodal large model may include operations S110 to S150.

[0031] In operation S110 , a training data set is acquired, where the training data set includes a plurality of images.

[0032] In operation S120, for each image, the GPS coordinate features of the image are represented by the GPS coordinates of the iso-global projection and encoded based on the random Fourier features to obtain the GPS coordinate feature vector of the image.

[0033] In operation S130, for each image, image-text collaborative encoding is performed on the image to obtain a collaborative encoding feature vector generated by fusing the image feature and the text feature of the image.

[0034] In operation S140, the GPS coordinate feature vector and the collaborative coding feature vector are input into the large model, and based on the embedding similarity of the GPS coordinate feature vector and the embedding similarity of the collaborative coding feature, multimodal comparative training is performed on the large model to obtain a trained large model.

[0035] In operation S150, the image to be positioned is input into the trained large model, and the longitude coordinate value and the latitude coordinate value of the image to be positioned are output.

[0036] In this embodiment, a multimodal aligned heuristic image location positioning method (Image Geographic Location based on Mulit-CLIP, IGeo-MCLIP) can be used to optimize the training of the large model by calculating the matching image information and geographic location information.

[0037] The global image geolocation task can be expressed as a mathematical model, that is, given a training dataset , N represents the number of images in the training data set, which contains three pairs of information, namely: image information , text information And the corresponding GPS coordinate information The goal of optimizing the model is to use a given training dataset The samples are used to train a global image geolocation recognition model W, and then the trained model W is used to predict the test dataset Specifically, the goal of the algorithm is to accurately predict the latitude and longitude of the query image given an image that the model has never seen. ,in, , the images in the query set can belong to any geographical location on the earth, K represents the number of images in the test dataset, Q is the representation of the training dataset, and W is the trained model.

[0038] In this embodiment, the multiple images in the training data may be multiple images collected in advance at different locations around the world or containing different contents.

[0039] This training method directly optimizes the similarity between image and text features and GPS coordinate features by introducing multimodal co-alignment, reduces positioning errors, overcomes the limitations of predefined categories, and improves positioning accuracy. In addition, through multimodal alignment and contrastive learning, the dependence on a large amount of training data is reduced, so that the model can achieve high positioning accuracy even with limited training data.

[0040] Based on the above embodiment, in operation S120, the GPS coordinates of the image are represented by the GPS coordinates of the equal earth projection, including: standardizing the projection of the GPS coordinates of the image and minimizing the distortion inherent in the standard system to obtain the projection coordinates of the GPS coordinates; scaling the projection coordinates to obtain the scaled projection coordinates.

[0041] In this embodiment, the encoding of the GPS coordinates of the image is an important part of realizing the multimodal feature alignment mapping. The main difficulty is that on the one hand, there is a lack of a dedicated GPS2Vec model, which makes it impossible to achieve high-precision mapping of GPS coordinates. On the other hand, since the GPS coordinate values ​​are relatively close, although the geographical locations represented by different GPS coordinates vary greatly, they are only slight differences for the neural network and cannot be mapped into feature vectors with the same differentiation. In order to effectively convert the 2D GPS coordinates (i.e. ) is encoded into a high-dimensional feature vector representation ,in The algorithm proposes a GPS coordinate representation technology of equal-earth projection encoded by random Fourier features, the core of which is to standardize the projection of GPS coordinates and minimize the distortion inherent in the standard system. Through the equal-earth projection technology, the algorithm offsets the distortion and ensures a more accurate representation of the coordinates on the earth's surface. That is, given , the feature encoding is calculated by the following formula:

[0042]

[0043]

[0044] in:

[0045] , ,

[0046]

[0047] After using the equal earth projection technology, the algorithm scales the GPS longitude to between (-1,1) and the latitude is scaled accordingly. Represents the longitude in GPS coordinates, Represents the longitude in encoded GPS coordinates, It represents the latitude in the encoded GPS coordinates, and θ represents the angle from the projection point to the projection center when the earth is projected.

[0048] On the basis of the above-mentioned embodiment, the GPS coordinates of the image are encoded based on the random Fourier features, including: constructing a coding layer, determining the frequency range required for the coding of the coding layer by presetting a fixed matrix, wherein each element in the fixed matrix obeys a Gaussian distribution; determining different coding frequencies in the random Fourier features based on the range of the element standard deviation; encoding different scale information of the projection coordinates based on different coding frequencies to obtain hierarchical features at different scales; transmitting the hierarchical features through a feedforward neural network, wherein the activation function of each feedforward neural network processes the Fourier coded features separately and generates a corresponding result feature vector; and combining multiple result feature vectors according to row elements to obtain a GPS coordinate feature vector.

[0049] In this embodiment, in order to obtain richer high-dimensional features from low-dimensional GPS coordinates, the algorithm applies the random Fourier-based position encoding technology to Specifically, we first construct a coding layer to determine the frequency range to be encoded. The algorithm uses a fixed matrix R to limit the frequency, whose elements are sampled from a Gaussian distribution with a standard deviation of After the matrix R is initialized, it remains unchanged throughout the training process, equal to the earth projection operator Projecting GPS coordinates Convert to , where the elements in the mth row and nth column follow a Gaussian distribution .

[0050] In order to construct a hierarchical feature representation that can capture information of different scales, the algorithm sets different change frequencies in the random Fourier features to obtain multiple high-dimensional representation features with different coarse and fine granularity. The range of values, therefore, the algorithm obtains the M-level hierarchy by the following calculation value:

[0051]

[0052] Allocation by index Value method, position encoder It is capable of encoding 2D GPS coordinate values ​​of different resolutions. In other words, this hierarchical setting enables the position encoder to capture the features of a specific location at different scales in a directionally controlled manner, which enhances the model's ability to efficiently learn a rich understanding of spatial information.

[0053] The encoded hierarchical features are then passed through the feedforward neural network MLP, where the activation function of each MLP is Independently process Fourier coded features and generate corresponding vectors Then, these result vectors are combined row by row to form a composite vector The calculation process of the position encoder is summarized as follows:

[0054]

[0055] On the basis of the above embodiment, operation S130 performs collaborative image-text encoding on the image to obtain a collaboratively encoded feature vector generated by fusing the image features and text features of the image, including: extracting the text title and text content from the image to obtain text features; using a pre-trained text encoder to encode the text features into a text feature vector; using a pre-trained image encoding model to encode the image features of the image into an image feature vector; using an attention mechanism to fuse the text feature vector and the image feature vector to obtain a collaboratively encoded feature vector.

[0056] In this embodiment, for each image, the text title of the image in the training data can be extracted, and then the OCR (Optical Character Recognition) engine is used to recognize the text content of each image, where the OCR engine provides font flipping, position correction and other functions, and then the detected text paragraphs are spliced ​​together. The pre-trained text encoder BERT is then used to encode the recognized text into a text feature vector , for the case where the text cannot be recognized by OCR in the image, that is, the encoded text feature vector is a diagonal matrix of the same dimension. Similarly, the pre-trained ResNeXt image coding model is used to encode the image into the corresponding image feature vector Finally, the attention mechanism is used to jointly generate the collaborative encoding feature vector by combining the image features and text features. .

[0057] Based on the above embodiment, in operation S140, multimodal contrast training is performed on the large model based on the embedding similarity of the GPS coordinate feature vector and the co-encoded feature, which may include: using contrast learning to align the similarity between the image features, text features and corresponding GPS coordinate features of the image, and minimizing the similarity between the GPS coordinate features of other images.

[0058] On the basis of the above embodiment, in operation S140, multimodal comparative training is performed on the large model based on the embedding similarity of the GPS coordinate feature vector and the collaborative coding feature, and also includes: dividing the training data set into multiple batches of training data; for each image in each batch of training data, constructing a set of randomly enhanced data sets, and the randomly enhanced data sets include at least one image negative sample; for each image in each batch of training data, injecting random noise into the GPS coordinates of each image, and the random noise is randomly sampled from a Gaussian distribution whose standard deviation is the standard deviation of the elements in the fixed matrix.

[0059] In this embodiment, a contrastive learning scheme is used to train the optimization model, where the optimization goal is to align the images. and text The features and their corresponding GPS coordinate features The similarity between them is minimized and its similarity with other GPS coordinates in the batch data is measured using a similarity function.

[0060] For example, for the same training batch of data, each image in the batch constructs a set of randomly enhanced datasets , which contains m negative image samples , in order to improve the quality of learning representation and promote the generalization and robustness of the model. For each set of data, the algorithm synchronously injects random noise of the coordinates into it To change its corresponding GPS coordinates This helps achieve spatially optimized smoothing in position encoders. From the standard deviation The additional noise-augmented position encoder learns a more robust representation that is less sensitive to slight changes in GPS.

[0061] For samples in the same batch, set the training temperature , the overall goal of model optimization is to minimize the loss function :

[0062]

[0063] The batch size of the model is , represents the size of the enhanced sample, is the fusion feature of image and text, Embedded vector features of GPS coordinates encoded for the model. Embedded vector features of GPS coordinates generated after adding noise.

[0064] By modeling the image geolocation task as a continuous function, the algorithm sequentially and directly compares the similarity between the embeddings of the query image and text and the embeddings of arbitrary GPS coordinate locations, effectively removing the constraint of recognition within predefined categories.

[0065] Based on the above embodiment, in operation S150, the image to be located is input into the trained large model, and the longitude coordinate value and the latitude coordinate value of the image to be located are output, which may include:

[0066] The image to be located is input into the trained large model, and the GPS coordinate feature vector corresponding to the image to be located is output;

[0067] The GPS coordinate feature vector is inverted to obtain the longitude and latitude coordinate values ​​of the image to be located.

[0068] In this embodiment, after the multimodal collaborative training large model is used to obtain a trained large model, the vectors of the image, text, and geographic location coordinates can be aligned and mapped based on the large model, and a continuous, high-dimensional, and complex mapping function is established between the image-text features and the GPS coordinate features, that is, given an image, the large model can automatically obtain its image and text feature vectors, and then compare the GPS embedding vectors that are similar to them. Finally, the corresponding accurate longitude and latitude coordinate values ​​are inverted and calculated.

[0069] Exemplarily, given an image of "Tour Group B in Region A, Tour Group C in Region A", the image is input into the large model of multimodal collaborative training. The model first automatically obtains the text element features: ["Travel around Region A", "Multi-region Highlights Tour of Region A", "Tour Group B", "Tour Group C"], and then encodes it into a text feature vector [[0.1234, -0.4567, 0.2345, -0.6789, ...], [0.3456, -0.1234, 0.5678, -0.2345, ...],[0.4567, -0.3456, 0.6789, -0.4567, ...], [0.2345, -0.5678, 0.4567, -0.6789,...]].

[0070] Then the large model automatically obtains the image element features: the high-dimensional image vector features encoded by the ResNeXt model are [[0.1234, -0.4567, 0.2345, 0.6789, -0.3456, 0.5678, 0.1987, -0.4321, ...],0.3456, 0.7890, -0.2345, 0.6543, 0.2109, -0.4987, 0.3671, 0.8012, ...],-0.1765, 0.5432, 0.2890, 0.6789, -0.4321, 0.5678, 0.1234, -0.4567, ...],0.2345, 0.6789, -0.3456, 0.5678, 0.1987, -0.4321, 0.3456, 0.7890]]. It should be noted that this high-dimensional vector contains multiple elements to capture rich visual features in the image, such as color, texture, shape, edge, texture and other information. Compared with low-dimensional vectors, this high-dimensional vector can better preserve the details and semantic information of the image.

[0071] Finally, the large model calculates similar high-dimensional vectors based on the acquired text feature vectors and image feature vectors, and inversely calculates the corresponding longitude and latitude coordinates, and it can be concluded that the photo was taken at a specific location.

[0072] It should be noted that when geolocating an image, there is no need to repeatedly train the model. After the large model training is completed, subsequent fine-tuning and optimization can be performed.

[0073] The image geolocation method based on a multimodal large model of the embodiment of the present invention combines image and text information, not only utilizing visual features in the image, but also utilizing text information (such as image title, text recognized by OCR, etc.) to enhance the accuracy of positioning. This multimodal combination enables the approximate location of the image to be inferred through text information even in the absence of explicit landmarks.

[0074] In summary, the method provided by the embodiment of the present invention can solve the following problems:

[0075] Solving the problem of lack of unique landmarks or place name cues: Traditional geolocation methods rely on explicit landmarks or place name cues in images, which do not work well in images that lack these features. IGeo-MCLIP enhances the accuracy of localization by combining image and text information, not only using visual features in the image, but also using text information (such as image captions, OCR-recognized text, etc.). This multimodal combination makes it possible to infer the approximate location of an image through text information even in the absence of explicit landmarks.

[0076] Solve global positioning problems: Traditional geolocation methods are usually limited to specific regions, while IGeo-MCLIP is trained on a global dataset and can process images from anywhere in the world. This method does not rely on predefined geographic categories, but directly compares the similarity of image and text features with GPS coordinate features through contrastive learning, thereby achieving accurate positioning on a global scale.

[0077] Solve the problem of maintaining a large-scale image library: Maintaining a library containing all possible images in the world is not feasible in practice. IGeo-MCLIP does not need to maintain a huge image library by using classification-based methods and contrastive learning. Instead, it trains models to learn the complex relationship between images, text, and geographic locations, thereby achieving efficient geolocation without the need for a large-scale image library.

[0078] Solving the problem of positioning error and training data requirements: Traditional classification-based methods may cause positioning errors due to images being far away from their category centers. IGeo-MCLIP reduces this error by directly optimizing the similarity between image and text features and GPS coordinate features through contrastive learning. In addition, IGeo-MCLIP reduces the reliance on large amounts of training data through multimodal alignment and contrastive learning, allowing the model to achieve high positioning accuracy even with limited training data.

[0079] The present invention also provides an image geolocation device based on a multimodal large model. Figure 3 The device is described in detail.

[0080] Figure 3 The structural block diagram of an image geo-positioning device based on a multimodal large model according to an embodiment of the present invention is schematically shown.

[0081] like Figure 3 As shown, the image geo-positioning device 300 based on the multimodal large model of this embodiment includes an acquisition module 310 , a first encoding module 320 , a second encoding module 330 , a training module 340 and an inference module 350 .

[0082] An acquisition module 310 is used to acquire a training data set, where the training data set includes a plurality of images;

[0083] The first encoding module 320 is used to perform, for each image, GPS coordinate representation of the GPS coordinate features of the image using iso-earth projection and encoding based on random Fourier features to obtain a GPS coordinate feature vector of the image.

[0084] The second encoding module 330 is used to perform image-text collaborative encoding on each image to obtain a collaborative encoding feature vector generated by fusing the image feature and the text feature of the image.

[0085] The training module 340 is used to input the GPS coordinate feature vector and the collaborative coding feature vector into the large model, and perform multimodal comparative training on the large model based on the embedding similarity of the GPS coordinate feature vector and the embedding similarity of the collaborative coding feature to obtain a trained large model.

[0086] The inference module 350 is used to input the image to be located into the trained large model and output the longitude coordinate value and latitude coordinate value of the image to be located.

[0087] According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits, or at least part of the functions of any one of them can be implemented in one module. According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits can be split into multiple modules for implementation. According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits can be at least partially implemented as hardware circuits, such as field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), systems on chips, systems on substrates, systems on packages, application specific integrated circuits (ASICs), or can be implemented by hardware or firmware in any other reasonable way of integrating or packaging the circuit, or by any one of the three implementation methods of software, hardware, and firmware, or by a proper combination of any of them. Alternatively, according to the embodiments of the present invention, one or more of the modules, submodules, units, and subunits can be at least partially implemented as computer program modules, and when the computer program modules are run, the corresponding functions can be executed.

[0088] For example, any multiple of the acquisition module 310, the first encoding module 320, the second encoding module 330, the training module 340 and the reasoning module 350 can be combined in one module / unit / sub-unit for implementation, or any one of the modules / units / sub-units can be split into multiple modules / units / sub-units. Alternatively, at least part of the functions of one or more of these modules / units / sub-units can be combined with at least part of the functions of other modules / units / sub-units and implemented in one module / unit / sub-unit. According to an embodiment of the present invention, at least one of the acquisition module 310, the first encoding module 320, the second encoding module 330, the training module 340 and the reasoning module 350 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or by any one of the three implementation methods of software, hardware and firmware or by a suitable combination of any of them. Alternatively, at least one of the acquisition module 310, the first encoding module 320, the second encoding module 330, the training module 340 and the reasoning module 350 may be at least partially implemented as a computer program module, which may perform a corresponding function when executed.

[0089] It should be noted that the device part in the embodiment of the present invention corresponds to the method part in the embodiment of the present invention, and the specific implementation details are also the same, which will not be repeated here.

[0090] Figure 4 A block diagram of an electronic device suitable for implementing the method described above according to an embodiment of the present invention is schematically shown. Figure 4 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present invention. Figure 4 As shown, the electronic device 400 according to an embodiment of the present invention includes a processor 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage part 408 to a random access memory (RAM) 403. The processor 401 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 401 may also include an onboard memory for caching purposes. The processor 401 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0091] In RAM 403, various programs and data required for the operation of electronic device 400 are stored. Processor 401, ROM 402 and RAM 403 are connected to each other via bus 404. Processor 401 performs various operations of the method flow according to the embodiment of the present invention by executing the programs in ROM 402 and / or RAM 403. It should be noted that the program can also be stored in one or more memories other than ROM 402 and RAM 403. Processor 401 can also perform various operations of the method flow according to the embodiment of the present invention by executing the programs stored in one or more memories.

[0092] According to an embodiment of the present invention, the electronic device 400 may further include an input / output (I / O) interface 405, which is also connected to the bus 404. The electronic device 400 may further include one or more of the following components connected to the I / O interface 405: an input portion 406 including a keyboard, a mouse, etc.; an output portion 407 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 408 including a hard disk, etc.; and a communication portion 409 including a network interface card such as a LAN card, a modem, etc. The communication portion 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the I / O interface 405 as needed. A removable medium 411, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 410 as needed, so that a computer program read therefrom is installed into the storage portion 408 as needed.

[0093] According to an embodiment of the present invention, the method flow according to an embodiment of the present invention can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program contains a program code configured to execute the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 409, and / or installed from the removable medium 411. When the computer program is executed by the processor 401, the above-mentioned functions defined in the system of the embodiment of the present invention are executed. According to an embodiment of the present invention, the system, equipment, device, module, unit, etc. described above can be implemented by a computer program module.

[0094] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiment; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present invention is implemented.

[0095] According to an embodiment of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, apparatus or device.

[0096] For example, according to an embodiment of the present invention, the computer-readable storage medium may include the ROM 402 and / or the RAM 403 described above and / or one or more memories other than the ROM 402 and the RAM 403 .

[0097] When the computer program is executed by the processor 401, the above functions defined in the system / device of the embodiment of the present invention are executed. According to the embodiment of the present invention, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0098] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 409, and / or installed from the removable medium 411. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0099] According to an embodiment of the present invention, the program code configured to execute the computer program provided by the embodiment of the present invention can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0100] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram may represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions configured to implement the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box may also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions. It will be understood by those skilled in the art that the features recorded in the various embodiments of the present invention can be combined and / or combined in various ways, even if such a combination or combination is not explicitly recorded in the present invention. In particular, without departing from the spirit and teachings of the present invention, the features recorded in the various embodiments of the present invention can be combined and / or combined in various ways. All these combinations and / or combinations fall within the scope of the present invention.

[0101] The above specific embodiments further illustrate the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An image geolocation method based on a multimodal large model, characterized in that: include: Acquire a training data set, wherein the training data set includes a plurality of images; For each image, the GPS coordinate features of the image are characterized by GPS coordinates of equal earth projection and encoded based on random Fourier features to obtain a GPS coordinate feature vector of the image; For each image, image-text collaborative encoding is performed on the image to obtain a collaborative encoding feature vector generated by fusing the image feature and the text feature of the image; Inputting the GPS coordinate feature vector and the collaborative coding feature vector into a large model, and performing multimodal comparative training on the large model based on the embedding similarity of the GPS coordinate feature vector and the embedding similarity of the collaborative coding feature to obtain a trained large model; The image to be located is input into the trained large model, and the longitude coordinate value and latitude coordinate value of the image to be located are output.

2. The method according to claim 1, characterized in that: The GPS coordinates of the image are represented by iso-geometry GPS coordinates, including: Standardizing the projection of the GPS coordinates of the image and minimizing the distortion inherent in the standard system to obtain the projection coordinates of the GPS coordinates; The projection coordinates are scaled to obtain scaled projection coordinates.

3. The method according to claim 2, characterized in that The GPS coordinates of the image are encoded based on random Fourier features, including: Constructing a coding layer, and determining a frequency range required for coding the coding layer by presetting a fixed matrix, wherein each element in the fixed matrix obeys a Gaussian distribution; determining different encoding frequencies in the random Fourier feature based on a range of the element standard deviation; Based on different encoding frequencies, different scale information of the projection coordinates is encoded to obtain hierarchical features at different scales; Passing the hierarchical features through a feed-forward neural network, wherein each activation function of the feed-forward neural network independently processes the Fourier encoded features and produces a corresponding resulting feature vector; The multiple result feature vectors are combined according to row elements to obtain the GPS coordinate feature vector.

4. The method according to claim 1, characterized in that: The step of performing image-text collaborative coding on the image to obtain a collaborative coding feature vector generated by fusing the image feature and the text feature of the image includes: Extracting text title and text content from the image to obtain the text features; Encoding the text features into text feature vectors using a pre-trained text encoder; Encoding the image features of the image into an image feature vector using a pre-trained image coding model; The text feature vector and the image feature vector are fused using an attention mechanism to obtain the collaboratively encoded feature vector.

5. The method according to claim 1, characterized in that The multimodal comparative training of the large model based on the embedding similarity of the GPS coordinate feature vector and the collaborative coding feature includes: By using contrastive learning, the similarities between the image features, text features and corresponding GPS coordinate features of the image are aligned, and the similarity between the GPS coordinate features of another image is minimized.

6. The method according to claim 5, characterized in that The multimodal comparative training of the large model based on the embedding similarity of the GPS coordinate feature vector and the collaborative coding feature also includes: Dividing the training data set into multiple batches of training data; For each image in each batch of training data, construct a set of randomly enhanced data sets, wherein the randomly enhanced data sets include at least one image negative sample; For each image in each batch of training data, random noise is injected into the GPS coordinates of each image, where the random noise is randomly sampled from a Gaussian distribution with a standard deviation equal to the standard deviation of the elements in the fixed matrix.

7. The method according to claim 1, characterized in that The step of inputting the image to be located into the trained large model and outputting the longitude coordinate value and the latitude coordinate value of the image to be located comprises: Input the image to be located into the trained large model to obtain the GPS coordinate feature vector corresponding to the image to be located; The GPS coordinate feature vector is inverted to obtain the longitude coordinate value and the latitude coordinate value of the image to be located.

8. An image geolocation device based on a multimodal large model, characterized in that: include: An acquisition module, used to acquire a training data set, wherein the training data set includes a plurality of images; A first encoding module is used for, for each image, performing GPS coordinate representation of the GPS coordinate feature of the image in an iso-geometry projection and encoding based on a random Fourier feature to obtain a GPS coordinate feature vector of the image; The second encoding module is used to perform image-text collaborative encoding on each image to obtain a collaborative encoding feature vector generated by fusing the image feature and the text feature of the image; A training module, used for inputting the GPS coordinate feature vector and the collaborative coding feature vector into a large model, and performing multimodal comparative training on the large model based on the embedding similarity of the GPS coordinate feature vector and the embedding similarity of the collaborative coding feature, so as to obtain a trained large model; The inference module is used to input the image to be located into the trained large model and output the longitude coordinate value and latitude coordinate value of the image to be located.

9. An electronic device, comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: A computer program or instruction is stored thereon, and when the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images