Geographic location recognition method, model training method and device for street view images

Abstract

Disclosed are a geographic location recognition method, a model training method and a device for street view images, wherein the training method is used to train a multimodal model based on contrastive learning, and the training method includes: using a model to obtain image feature vectors of street view image samples in an image sample set of a geographic area; using a model to obtain text feature vectors of text labels of street view image samples of multiple geographic areas in a label sample set; using a model to combine text feature vectors with street view fusion features to obtain multimodal fusion features of multiple geographic areas; using a model to obtain similarities between image feature vectors and multimodal fusion features of multiple geographic areas respectively; performing loss calculation based on the similarity and the geographic area labels of the street view image samples, and adjusting model parameters based on the loss calculation results. The technical solution provided by the present disclosure is conducive to improving the training efficiency of the model, enriching the implementation methods of geographic location recognition, and improving the accuracy of geographic location recognition.

Description

Technical Field

[0001] The present disclosure relates to computer vision technology, and more particularly to a method and device for identifying geographic location of street view images, a method and device for training a model, a storage medium, and an electronic device. Background Art

[0002] Using street view images for geographic location recognition can not only play an important role in applications such as navigation and location services, but also provide important data support for applications such as urban planning, intelligent transportation and geographic information systems. At the same time, it is also of great significance in geographic information security research. However, how to efficiently and accurately identify the geographic location of street view images is a challenging topic. Summary of the invention

[0003] In order to solve the above technical problems, the present disclosure is proposed. The embodiments of the present disclosure provide a method and device for identifying geographic location of a street view image, a method and device for training a model, a storage medium and an electronic device.

[0004] According to a first aspect of an embodiment of the present disclosure, a model training method is provided, which is used to train a multimodal model based on contrastive learning, and the training method includes: using the multimodal model to obtain image feature vectors of street view image samples in an image sample set of a geographical area, wherein the geographical area includes: at least four area ranges, and the at least four area ranges decrease step by step; using the multimodal model to obtain text feature vectors of text labels of street view image samples of multiple different geographical areas in a label sample set; using the multimodal model to combine the text feature vectors of the text labels of the street view image samples of the multiple geographical areas with the street view fusion features of the multiple geographical areas respectively, to obtain the multimodal fusion features of the multiple geographical areas; using the multimodal model to obtain the similarity between the image feature vector and the multimodal fusion features of the multiple geographical areas respectively; performing loss calculation according to the similarity and the geographical area label of the street view image sample, and adjusting the parameters of the multimodal model according to the result of the loss calculation; wherein the successfully trained multimodal model based on contrastive learning is used to determine the geographical location of the street view image to be identified.

[0005] According to a second aspect of an embodiment of the present disclosure, a method for identifying a geographic location of a street view image is provided, and the method comprises: providing a street view image to be identified as an input image to a multimodal model based on contrastive learning, and obtaining an image feature vector of the street view image to be identified; obtaining text feature vectors of text labels of street view images of multiple geographic areas, wherein the geographic area comprises: at least four area ranges, and the at least four area ranges decrease step by step; the text feature vectors of text labels of street view images of the multiple geographic areas and street view fusion features of the multiple geographic areas are provided to the multimodal model, and multimodal fusion features of the multiple geographic areas are obtained by combining the text feature vectors of text labels of street view image samples of the multiple geographic areas with the street view fusion features of the multiple geographic areas respectively through the multimodal model; the image feature vector and the multimodal fusion features of the multiple geographic areas are provided to the multimodal model, and similarity calculation is performed through the multimodal model to obtain multiple similarities; and the geographic location of the street view image to be identified is determined according to the multiple similarities.

[0006] According to a third aspect of an embodiment of the present disclosure, a model training device is provided, which is used to train a multimodal model based on contrastive learning, and the model training device includes: a first module, which is used to use the multimodal model to obtain image feature vectors of street view image samples in an image sample set of a geographical area, wherein the geographical area includes: at least four area ranges, and the at least four area ranges decrease step by step; a second module, which is used to use the multimodal model to obtain text feature vectors of text labels of street view image samples in multiple different geographical areas in a label sample set; a third module, which is used to use the multimodal model to The text feature vectors of the text labels of the street view image samples of the multiple geographical areas are respectively combined with the street view fusion features of the multiple geographical areas to obtain the multimodal fusion features of the multiple geographical areas; the fourth module is used to use the multimodal model to obtain the similarity between the image feature vector and the multimodal fusion features of the multiple geographical areas; the loss module is used to calculate the loss according to the similarity and the geographical area labels of the street view image samples, and adjust the parameters of the multimodal model according to the results of the loss calculation; wherein the successfully trained multimodal model based on contrastive learning is used to determine the geographical location of the street view image to be identified.

[0007] According to a fourth aspect of an embodiment of the present disclosure, a device for identifying a geographic location of a street view image is provided, the device comprising: a fifth module, for providing a street view image to be identified as an input image to a multimodal model based on contrastive learning, and obtaining an image feature vector of the street view image to be identified; a sixth module, for obtaining text feature vectors of text labels of street view images of multiple geographical areas, wherein the geographical areas include: at least four area ranges, and the at least four area ranges decrease step by step; a seventh module, for providing the text feature vectors of the text labels of the street view images of the multiple geographical areas and the street view fusion features of the multiple geographical areas to the multimodal model, and obtaining, through the multimodal model, multimodal fusion features of the multiple geographical areas formed by combining the text feature vectors of the text labels of the street view images of the multiple geographical areas with the street view fusion features of the multiple geographical areas; an eighth module, for providing the image feature vector and the multimodal fusion features of the multiple geographical areas to the multimodal model, and performing similarity calculation through the multimodal model to obtain multiple similarities; and a recognition module, for determining the geographic location of the street view image to be identified according to the multiple similarities.

[0008] According to a fifth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and the computer program is used to implement the above method.

[0009] According to the sixth aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a processor; a memory for storing instructions executable by the processor; the processor is configured to read the executable instructions from the memory and execute the instructions to implement the above method.

[0010] Based on the above-mentioned embodiments of the present disclosure, a geographic location recognition method and device for street view images, a model training method and device, a storage medium and an electronic device are provided. By using street view image samples and text labels of street view image samples for model training, the input required for training is an image-text label pair, and the annotations of street view image samples and the text labels of street view image samples are only related to the geographic area and have nothing to do with the image content. In particular, the number of text labels of street view image samples is the same as the number of geographic areas, thereby greatly simplifying the labeling of samples. Since a multimodal model based on contrastive learning is usually a model that has been pre-trained using a large number of sample pairs, the multimodal model that needs to be trained in the present disclosure usually already has a good recognition foundation, so when training it in a vertical field, a limited number of sample pairs can be used. A good training effect can be achieved with a number of sample pairs; since the street view fusion features of a geographical area in the present disclosure can more clearly reflect the common features of the geographical area, and the text feature vectors of the text labels of the street view image samples of different geographical areas often have more obvious differences, therefore, the multimodal fusion features of multiple geographical areas formed by combining the text feature vectors of the text labels of the street view image samples of multiple geographical areas with the street view fusion features of multiple geographical areas respectively can often better highlight the uniqueness of each geographical area, which is conducive to eliminating the influence of interference factors and other contents in the sample on the recognition, and is conducive to avoiding the difficulties in the recognition of geographical areas caused by the similar image content in a small area, so as to enable the model to converge quickly during the training process, and is conducive to improving the recognition performance of the successfully trained model. In addition, by representing the geographical area with at least four gradually reduced regional ranges, it is conducive to further improving the ability of the successfully trained model to accurately locate in a small range; it can be seen that the technical solution provided by the present disclosure is conducive to improving the training efficiency of the model and enriching the implementation method of geographical location recognition, while improving the accuracy of geographical location recognition.

[0011] The technical solution of the present disclosure is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] By describing the embodiments of the present disclosure in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present disclosure will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and do not constitute a limitation of the present disclosure. In the drawings, the same reference numerals generally represent the same components or steps.

[0013] Figure 1 A schematic diagram of the structure of a multimodal model based on contrastive learning disclosed in the present invention;

[0014] Figure 2A flowchart of an embodiment of a model training method disclosed herein;

[0015] Figure 3 A schematic diagram of an embodiment of a model training method disclosed herein;

[0016] Figure 4 A flowchart of an embodiment of obtaining street view fusion features of each geographical area disclosed in the present invention;

[0017] Figure 5 A flowchart of an embodiment of obtaining multimodal fusion features of each geographical area according to the present disclosure;

[0018] Figure 6 A flowchart of an embodiment of a method for identifying a geographic location of a street view image disclosed in the present invention;

[0019] Figure 7 A schematic diagram of an embodiment of a method for identifying a geographic location of a street view image disclosed in the present invention;

[0020] Figure 8 It is a structural schematic diagram of an embodiment of a model training device disclosed in the present invention;

[0021] Fig. 9 It is a structural schematic diagram of an embodiment of a geographical location recognition device for a street view image disclosed in the present invention;

[0022] Fig.10 is a structural diagram of an electronic device provided by an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0023] The exemplary embodiments according to the present disclosure will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described here.

[0024] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.

[0025] Those skilled in the art can understand that the terms "first" and "second" in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, etc., and neither represent any specific technical meaning nor indicate the necessary logical order between them.

[0026] It should also be understood that in the embodiments of the present disclosure, “plurality” may refer to two or more than two, and “at least one” may refer to one, two, or more than two.

[0027] It should also be understood that any component, data or structure mentioned in the embodiments of the present disclosure can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.

[0028] In addition, the term "and / or" in this disclosure is only a description of the association relationship of associated objects, indicating that there may be three relationships, such as A and / or B, which can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this disclosure generally indicates that the associated objects before and after are in an "or" relationship.

[0029] It should also be understood that the description of the various embodiments in the present disclosure focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced to each other, and for the sake of brevity, they will not be described one by one.

[0030] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0031] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0032] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0033] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0034] The embodiments of the present disclosure can be applied to electronic devices such as terminal devices, computer systems, servers, etc., which can operate with many other general or special computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments and / or configurations suitable for use with electronic devices such as terminal devices, computer systems or servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, small computer systems, large computer systems, and distributed cloud computing technology environments including any of the above systems, etc.

[0035] Electronic devices such as terminal devices, computer systems, servers, etc. can be described in the general context of computer system executable instructions (such as program modules) executed by computer systems. Generally, program modules can include routines, programs, target programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in a distributed cloud computing environment. In a distributed cloud computing environment, tasks can be performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can be located on local or remote computing system storage media including storage devices.

[0036] SUMMARY OF THE DISCLOSURE

[0037] In the process of realizing the present disclosure, the inventors found that the geographic location recognition of street view images is usually realized by image classification and recognition technology. For example, the geographic location recognition of street view images is realized by using traditional image classification technology, image retrieval technology or image embedding and retrieval technology. In a specific example, the input image is classified by using classification algorithms such as support vector machine and random forest, and the geographic location of the input image is located according to the classification results. In another specific example, a database containing a large number of labeled images is pre-established, and then the image most similar to the input image is found by using image retrieval technology, so as to determine the geographic location of the input image according to the label of the most similar image. In another specific example, the embedded vector of the input image is generated by a deep learning model, and then the similarity search is performed for the embedded vector in the pre-built embedded vector database, and the geographic location of the input image is determined according to the search results. However, the geographic location recognition effect of directly introducing the existing image classification and recognition technology into the geographic location recognition scenario of street view images is not ideal. The reasons include:

[0038] 1. The core subject of street view images is sometimes unclear, and the content presented in street view images often does not include clear subjects that can be single-classified.

[0039] 2. Street view images are usually more complex. For example, there may be occlusions (such as those caused by trees or vehicles) and dynamic content (such as pedestrians) in street view images, which will interfere with the recognition accuracy of the model.

[0040] 3. Different geographical locations sometimes have highly similar visual features, especially when locating a more precise small-scale geographical location. The smaller the area covered by the geographical location, the greater the similarity between a geographical location and the street scenes (such as buildings, streets, etc.) of its surrounding geographical locations. This is very unfavorable for image classification and recognition technology, and will affect the accuracy of geographical location recognition of street view images.

[0041] From the above description, it can be seen that due to the many problems with the content of street view images, the existing image classification and recognition technology only uses the content of street view images to realize the geographic location recognition of street view images, which inevitably has a great impact on the geographic location recognition effect. If in the process of geographic location recognition, in addition to considering the street view images to be identified, the common features in an area and other information that can effectively assist in recognition are also considered, it will be beneficial to improve the accuracy of geographic location recognition.

[0042] Exemplary Overview

[0043] The geographic location recognition technology of street view images disclosed in the present invention can be applied to a variety of scenarios. For example, the geographic location recognition technology of street view images provided in the present invention can be applied to location service scenarios. An example of the application of the geographic location recognition technology of street view images disclosed in the present invention in a location service scenario is described below.

[0044] Currently, multiple photos containing a target object (such as a missing person or a person to be investigated) are collected. The location service APP (Application) in the mobile phone or computer is opened, and according to the prompts of the location service APP, multiple street view photos are provided to the location service APP as street view images to be identified. The location service APP performs geographic location identification processing on each photo. The location service APP displays the geographic location information of each photo, such as a city, a district / county, a township / street, and an average location coordinate, etc., so that the activity situation of the target object over a period of time can be obtained.

[0045] Exemplary Methods

[0046] The geographic location recognition technology of street view images disclosed in the present invention is implemented by using a multimodal model based on contrastive learning. An example of the structure of the multimodal model is as follows: Figure 1 An example of a training method for the multimodal model is shown in Figure 2 and Figure 3 As shown below. Figure 1 , Figure 2 and Figure 3 The structure of the multimodal model based on contrastive learning and the training method thereof disclosed in the present invention are explained.

[0047] like Figure 1The multimodal model based on contrastive learning shown mainly includes: a text encoder 100, an image encoder 101, a feature merging module 102 and a similarity calculation module 103. In addition, the multimodal model may also optionally include: a sample feature fusion encoder 104. The multimodal model based on contrastive learning disclosed in the present invention may be an improved model based on the CLIP (Contrastive Language-Image Pre-training, based on pre-training of text-image pairs) model of OpenAI (OpenArtificial Intelligence). The specific improvements are described below.

[0048] like Figure 2 The method shown mainly includes: S200, S201, S202, S203 and S204. Figure 1 and Figure 3 right Figure 2 Each step in is described separately.

[0049] Figure 2 In S200, using a multimodal model, an image feature vector of a street view image sample in an image sample set of a geographical area is obtained.

[0050] The multimodal model in the present disclosure can be considered as a model trained using samples in more than one form. For example, a model trained using samples in the form of images and samples in the form of text. Figure 3 The lower left corner shows a plurality of street view image samples) which generally refer to images containing human features, such as images containing buildings, images containing roads, or images containing inscriptions / road signs, etc. Of course, street view image samples may also include: cars, pedestrians, flowers, plants, and trees, etc.

[0051] The geographical area in the present disclosure includes: at least four area ranges, and at least four area ranges decrease step by step. That is to say, one area range can cover another area range adjacent to it. For example, taking four area ranges as an example, these four area ranges may include: city, district / county, township / street and custom area; the city covers the district / county, and the city includes multiple districts / counties; the district / county covers the township / street, and a district / county includes multiple townships / streets; the township / street covers the custom area, and a township / street includes multiple custom areas. The custom area in the present disclosure can be represented by the coordinates of a position point in the predetermined area, such as a longitude and latitude. The predetermined area in the present disclosure can be a range formed by taking a predetermined point in the township / street area as the center of the circle and a predetermined earth spherical distance as the radius. The predetermined point can be set according to actual needs, for example, the coordinates of a certain street view image sample can be used as the predetermined point; for another example, the predetermined point is set in the township / street on the principle of using a reasonable number of circles as much as possible and covering the township / street without omission. It should be noted that the sizes of the predetermined areas of any two different geographical areas are usually the same. Of course, the present disclosure does not exclude the possibility that the sizes of the predetermined areas of any two different geographical areas may be different. The present disclosure sets a geographical area including four area ranges, and the largest area range is a city, and the smallest area range is a custom area formed by a center and a radius, so that the technology of the present disclosure is more suitable for application scenarios with high requirements for positioning accuracy and small requirements for positioning area.

[0052] The present disclosure may set multiple image sample sets, and one image sample set corresponds to one geographical area, and different image sample sets correspond to different geographical areas. That is, all street view image samples contained in one image sample set have the same text label, and the text label is used to describe the geographical area to which the street view image sample belongs. An example of a text label may be: an image located in a certain custom area of ​​a certain district / county / township / street in a certain city.

[0053] The present disclosure can divide all street view image samples into geographical regions, thereby forming an image sample set for each geographical region. This process can be implemented by the following steps a and b:

[0054] Step a: Calculate the distances of all street view image samples from the predetermined point on the earth's spherical surface based on the coordinates of all street view image samples and the coordinates of the predetermined point.

[0055] Specifically, each street view image sample in the present disclosure has coordinates (such as longitude and latitude), and the coordinates of different street view image samples may be the same or different. Each predetermined point in the present disclosure also has coordinates, and the coordinates of different predetermined points are different. In one example, the coordinates of the predetermined point may be the coordinates of a street view image sample. For any street view image sample, the present disclosure may calculate the earth's spherical distance between the point represented by the coordinates of the street view image sample and all predetermined points. The earth's spherical distance may be calculated using the following formula (1):

[0056] L AB =R·cos -1 (cos(A lat )cos(B lat )cos(B lon -A lon )+sin(A lat )sin(B lat )) Formula (1)

[0057] In the above formula (1), R is the radius of the earth, A lon ,A lat Indicates the latitude and longitude of point A; B lon ,B lat Indicates the latitude and longitude of point B; L AB Represents the distance on the earth's spherical surface between point A and point B.

[0058] Step b: determining the image sample sets of the geographical areas to which all the street view image samples belong according to the earth's spherical distance and the radius of each predetermined area.

[0059] Specifically, for any street view image sample, after obtaining the earth spherical distance between the point represented by the coordinates of the street view image sample and all the predetermined points by using step a, a minimum earth spherical distance can be selected, and the minimum earth spherical distance corresponds to a predetermined point. The radius of the corresponding predetermined point is compared with the minimum earth spherical distance. If the minimum earth spherical distance is not greater than the radius corresponding to the predetermined point, the street view image sample is classified into the image sample set of the geographical area to which the predetermined point belongs; if the minimum earth spherical distance is greater than the radius corresponding to the predetermined point, the street view image sample can be abandoned, that is, the street view image sample should not be classified into any image sample set.

[0060] It should be noted that, for any street view image sample, the present disclosure may not calculate the earth spherical distance between the point represented by the coordinates of the street view image sample and all the predetermined points, but calculate the earth spherical distance between the point represented by the coordinates of the street view image sample and the current predetermined point in sequence according to the arrangement order of the predetermined points, and each time a earth spherical distance is calculated, it is determined whether the earth spherical distance is not greater than the radius corresponding to the current predetermined point. If it is not greater than the radius corresponding to the current predetermined point, the street view image sample is divided into the image sample set of the geographical area to which the predetermined point belongs, and the geographical spherical distance is no longer calculated for the subsequent predetermined points. If it is greater than the radius corresponding to the current predetermined point, the earth spherical distance is continued to be calculated for the next predetermined point. Since there may be overlapping areas between two different predetermined areas, and the coordinates of the street view image sample may be located in the overlapping area, using the minimum earth spherical distance is conducive to classifying the street view image sample into the image sample set of the geographical area to which the predetermined area closest to the center of the circle belongs, and is conducive to making the street view image sample located in a more appropriate image sample set. The method of calculating and immediately executing the judgment operation is helpful in reducing the amount of calculation for model training.

[0061] After all street view image samples are divided into image sample sets of corresponding geographical areas, the present disclosure can generate description information of the custom area using the coordinates of the street view image samples in the image sample set. Specifically, for the image sample set of any geographical area, the coordinates of all street view image samples in the image sample set are averaged, and the result of the average calculation (such as the longitude mean and latitude mean of all street view image samples in the image sample set) is used as the coordinates for describing the custom area in the geographical area. An example of the geographical area of ​​the present disclosure can be expressed as: 30.2944 / 120.117206, Gudang Street, Xihu District, Hangzhou City.

[0062] In one example, the present disclosure may utilize an image encoder 101 to obtain image feature vectors of street view image samples in an image sample set of a geographic area, and the image encoder 101 may be an image encoder in a CLIP model (e.g., Figure 3 Specifically, a street view image sample in a set of image samples of a geographical area is provided as an input image to the image encoder 101, and the image encoder 101 outputs an image feature vector of the street view image sample. Since the CLIP model has been pre-trained with hundreds of millions of image sample and text sample pairs, the image feature vector is extracted by using the image encoder in the CLIP model, which is conducive to obtaining an image feature vector that can more accurately describe the street view image sample.

[0063] S201. Obtain text feature vectors of text labels of street view image samples of multiple different geographical areas in a label sample set using a multimodal model.

[0064] The label sample set disclosed in the present invention includes: street view image sample text labels of all geographical areas (such as Figure 3 The number of text labels contained in the label sample set is usually the number of geographical areas (such as Figure 3 , i.e., one geographical area corresponds to one text label, and the number of geographical areas is usually the total number of all custom areas. In one example, a text label may be: This is an image of 30.2944 / 120.117206, Gudang Street, Xihu District, Hangzhou City. 30.2944 / 120.117206 represents a custom area.

[0065] The present disclosure can classify all street view image samples into corresponding image sample sets, and after setting the coordinates of the custom area of ​​the geographic area according to the coordinates of all street view image samples in the image sample set, set a text label for each image sample set to form a text label set (such as Figure 3 n area street view labels in (2)).

[0066] The present disclosure can use a text encoder 100 to obtain text feature vectors of text labels of street view image samples of multiple different geographical areas in a label sample set. The text encoder 100 can be a text encoder in a CLIP model (e.g. Figure 3 The CLIP text encoder in the CLIP model), that is, the text encoder 100 may include: a word segmentation module in the CLIP model and a text feature extraction module in the CLIP model. Specifically, for the text label of the street view image sample of any geographical area in the label sample set, the word segmentation module can be used to perform word segmentation on the text label, and each word after the word segmentation process can be converted into vector data respectively; then, the text feature extraction module is used to perform feature extraction on the converted vector data, so as to obtain the text feature vector of the text label of the street view image sample of the geographical area. Since the CLIP model has been pre-trained with hundreds of millions of image sample text sample pairs, therefore, using the text encoder in the CLIP model to extract the text feature vector is conducive to obtaining a text feature vector that can more accurately describe the text label of the street view image sample.

[0067] In the training process of the multimodal model of the present invention, if the geographical area does not change, all street view image samples do not change, and the text labels of all street view image samples do not change, then the present invention can store all text feature vectors in the text feature vector set (i.e., store the text feature vectors of the text labels of the street view image samples of all geographical areas for the first time) when obtaining the text feature vectors of the text labels of the street view image samples of all geographical areas for the first time. Figure 3 T 1 , T 2 , T 3 , ..., and T n Stored in a text feature vector set), so that in the subsequent model training process, there is no need to use the text encoder 100 to perform the text feature vector extraction operation each time, and the text feature vectors of the street view image sample text labels of all geographical areas can be directly read from the text feature vector set, which is beneficial to reduce the computational complexity of model training and improve the efficiency of model training.

[0068] S202: Using a multimodal model, the text feature vectors of the text labels of the street view image samples of the plurality of geographical regions are respectively combined with the street view fusion features of the plurality of geographical regions to obtain the multimodal fusion features of the plurality of geographical regions.

[0069] The street view fusion feature of a geographical area in the present disclosure refers to a feature formed by fusing multiple street view image samples belonging to the same geographical area. The present disclosure can use the sample feature fusion encoder 104 to obtain the street view fusion feature of each geographical area. For example, for any geographical area, the present disclosure uses the sample feature fusion encoder 104 to perform feature fusion processing on all street view image samples in the image sample set of the geographical area, thereby generating the street view fusion feature of the geographical area.

[0070] In one example, the sample feature fusion encoder 104 disclosed in the present invention may include: an input layer, a deep convolutional neural network based on a residual network architecture with a classification layer discarded, and a mean calculation module. The input layer is mainly used to perform vectorization conversion processing on street view image samples. The deep convolutional neural network based on a residual network architecture is a network for implementing classification. For example, the deep convolutional neural network based on a residual network architecture with the classification layer removed can be: the residual network after removing the last classification layer in ResNet50 (Residual net, residual network), which can be recorded as MRestNet50. MRestNet50 is mainly used to perform feature extraction operations. The mean calculation module is mainly used to perform mean calculation processing on the extracted vectors.

[0071] In one example, the process of obtaining the street view fusion features of each geographical area by using the sample feature fusion encoder 104 in the present disclosure can be as follows: Figure 4 shown.

[0072] Figure 4 In S400, for any geographical area, a plurality of street view image samples in the image sample set of the geographical area are respectively provided as input images to the input layer in the sample feature fusion encoder 104, and the input layer converts the plurality of street view image samples into vector data respectively.

[0073] In one example, all street view image samples in the image sample set of the geographic area are provided to the input layer respectively, so that vector data of each street view image sample is obtained according to the output of the input layer.

[0074] When the number of geographical areas is n, the n image sample sets can be as follows: Figure 3 The area 1 classification street view picture sample set, the area 2 classification street view picture sample set, ..., and the area n classification street view picture sample set shown in (1).

[0075] S401. Vector data of multiple street view image samples are provided to MRestNet50 respectively, and image feature vectors of the multiple street view image samples are obtained through MRestNet50.

[0076] In one example, the image feature vector of any street view image sample obtained by MRestNet50 can be represented as a one-dimensional vector data The one-dimensional vector data may include 768 vector elements. The i in the vector represents the i-th street view image sample in the image sample set of the geographic area.

[0077] S402: Calculate the mean of each vector element in the image feature vector of the plurality of street view image samples through a mean calculation module to obtain the street view fusion feature of the geographical area.

[0078] In one example, the mean value calculation module may use the following formula (2) to calculate the mean value of each vector element:

[0079]

[0080] In formula (2), n represents the total number of street view image samples in the image sample set of the geographic area; represents the image feature vector of the i-th street view image sample (which can be one-dimensional vector data), The street view fusion features representing the geographic area, in one example, It is one-dimensional vector data containing 768 vector elements, and each vector element in the vector data is the average value of the vector elements at the corresponding position in the image feature vectors of the n street view image samples.

[0081] Due to the deep convolutional neural network based on residual network architecture (such as M RestNet50 ) can effectively solve the gradient vanishing problem in deep neural networks, and is usually a pre-trained neural network. Therefore, the present disclosure obtains the street view fusion features of the geographical area by utilizing the input layer, a deep convolutional neural network based on a residual network architecture with the classification layer discarded, and a mean calculation module, which is conducive to enabling the street view fusion features to better describe the features of street view image samples in the geographical area.

[0082] In the training process of the multimodal model of the present invention, if the geographical area does not change and the street view image samples in the image sample sets of all geographical areas do not change, the present invention can store the street view fusion features of all geographical areas in the street view fusion feature library FF (such as Figure 3 The street view fusion feature library FF shown in (1) includes street view fusion features of n geographical regions, namely, street view fusion features of region 1, street view fusion features of region 2, ... and street view fusion features of region n, namely Figure 3 FF 1 , FF 2 , FF 3 , ..., and FF n ), thus, in the subsequent model training process, it is not necessary to use the sample feature fusion encoder 104 to perform vectorization conversion, image feature extraction, mean calculation and other operations for each geographical area respectively each time, and the street view fusion features of all geographical areas can be directly read from the street view fusion feature library FF, which is beneficial to reduce the calculation amount of model training and improve the efficiency of model training.

[0083] In one example, the present disclosure can use the feature merging module 102 to combine the text feature vectors of the text labels of the street view image samples of multiple geographical regions with the street view fusion features of the multiple geographical regions, thereby obtaining the multimodal fusion features of the multiple geographical regions. That is, for any geographical region, the feature merging module 102 is used to combine the text feature vectors of the text labels of the street view image samples of the geographical region with the street view fusion features of the geographical region to obtain the multimodal fusion features of the geographical region.

[0084] The multimodal fusion feature disclosed in the present invention may refer to a feature fused by the features of the text modality and the features of the image modality. The features of these two modalities may be fused in a variety of ways, for example, they may be fused using an attention mechanism. Specifically, for any geographical area, the feature merging module 102 combines the text feature vector of the text label of the street view image sample of the geographical area with the street view fusion feature of the geographical area based on the attention mechanism, thereby obtaining the multimodal fusion feature of the geographical area.

[0085] By adding the attention mechanism in the process of generating multimodal fusion features, the combination process of text feature vectors and street view fusion features can focus on more significant features, so that the multimodal fusion features can better reflect the unique characteristics of the geographical area, thereby improving the model training efficiency and enhancing the location recognition ability of the successfully trained multimodal model.

[0086] In one example, the process of obtaining the multimodal fusion features of each geographical area by the feature merging module 102 in the present disclosure can be as follows: Figure 5 shown.

[0087] Figure 5 In S500, for any geographical area, according to a first weight, the text feature vector of the text label of the street view image samples of the geographical area is adjusted to obtain a first feature vector.

[0088] The first weight in the present disclosure is a weight set for the text feature vector, and the first weight can be used to control the importance of the text feature vector in the combination process. The first weight is usually an empirical value. The text feature vectors of the text labels of street view image samples in different geographical areas usually correspond to the same first weight. The present disclosure can perform a product operation on the first weight wt and the text feature vector, and use the result of the product operation as the first feature vector. The present disclosure can pre-set a group of first weights, and during the training process, the corresponding weight value can be selected from the group.

[0089] S501: Adjust the street view fusion feature of the geographical area according to the second weight to obtain a second feature vector.

[0090] The second weight in the present disclosure is a weight set for the street view fusion feature of a geographical area, and the second weight can be used to control the importance of the street view fusion feature of a geographical area in the combination process. The second weight is usually an empirical value. The street view fusion features of different geographical areas usually correspond to the same second weight. The present disclosure can perform a product operation on the second weight wff and the street view fusion feature of the geographical area, and use the result of the product operation as the second feature vector. The present disclosure can pre-set a group of second weights, and during the training process, the corresponding weight value can be selected from the group.

[0091] S502: Obtain a multimodal fusion feature of the geographic area according to the vector sum of the first feature vector and the second feature vector.

[0092] The vector sum operation of the present disclosure can be expressed in the form of the following formula (3):

[0093] T′i=(T i *wt)+(FFi *wff) Formula (3)

[0094] In the above formula (3), T′i represents the multimodal fusion features of the i-th geographical area (e.g. Figure 3 T 1 '、T 2 '、T 3 ', ..., and T n '); T i represents the text feature vector of the text label of the street view image sample of the ith geographical area, wt represents the weight set for the text feature vector, that is, the first weight, FFi represents the street view fusion feature of the ith geographical area, and wff represents the weight set for the street view fusion feature of the ith geographical area, that is, the second weight.

[0095] S203: Using a multimodal model, obtain similarities between image feature vectors of street view image samples and multimodal fusion features of multiple geographic areas.

[0096] The similarity in the present disclosure is used to indicate the possibility that the street view image sample belongs to a geographical area, and the similarity in the present disclosure can be expressed by the dot product calculation result. Specifically, the present disclosure can use the similarity calculation module 103 to perform similarity calculation, that is, the similarity calculation module 103 performs dot product calculation on the image feature vector of a street view image sample obtained in S200 and the multimodal fusion feature of each geographical area, so as to obtain multiple dot product calculation results, that is, to obtain multiple similarities, and one similarity corresponds to one geographical area, for example, Figure 3 For the first street view image sample, n similarities are obtained, namely, I 1 T 1 '、I 1 T 2 '、I 1 T 3 ', ..., and I 1 T n '; For the second street view image sample, obtain n similarities, i.e., I 2 T 1 '、I 2 T 2 '、I 2 T 3 ', ..., and I 2 T n ', for the third street view image sample, we obtain n similarities, i.e., I 3 T 1 '、I 3 T 2 '、I 3 T 3 ', ..., and I3 T n ', ... For the nth street view image sample, n similarities are obtained, i.e., I n T 1 '、I n T 2 '、I n T 3 ', ..., and I n T n '.

[0097] S204: performing loss calculation based on the above similarity and the geographic area labels of the street view image samples, and adjusting the parameters of the multimodal model based on the result of the loss calculation.

[0098] Specifically, the present disclosure can select the maximum similarity from all similarities, so as to obtain the geographical area corresponding to the maximum similarity. Since each street view image sample corresponds to a text label (for example, the image sample set to which the street view image sample belongs corresponds to a corresponding geographical area, and the geographical area corresponds to a corresponding text label), the loss function can be used to calculate the difference between the geographical area corresponding to the maximum similarity and the text label, and according to the calculation result, the back propagation algorithm is used to adjust the parameters in the multimodal model, such as adjusting the parameters in the text encoder 100 and the image encoder 101.

[0099] Repeat the above steps until the loss function converges, and the multimodal model is successfully trained. If all street view image samples are used and the convergence of the loss function still does not meet the requirements, the training process fails. The multimodal model can be trained again by adjusting the initial settings of the multimodal model, adjusting the street view image samples, etc., until the multimodal model is successfully trained. The successfully trained multimodal model is used to determine the geographic location of the street view image to be identified.

[0100] By using street view image samples and text labels of street view image samples for model training, the input required for training is an image-text label pair, and the annotations of street view image samples and the text labels of street view image samples are only related to the geographical area and have nothing to do with the image content. In particular, the number of text labels of street view image samples is the same as the number of geographical areas, thereby greatly simplifying the labeling of samples; since the multimodal model based on contrastive learning is usually a model that has been pre-trained with a large number of sample pairs, the multimodal model that needs to be trained in the present disclosure usually has a good recognition foundation, so when training it in a vertical field, a limited number of sample pairs can be used to achieve a good training effect; since a multimodal model based on contrastive learning is usually a model that has been pre-trained with a large number of sample pairs, the multimodal model that needs to be trained in the present disclosure usually has a good recognition foundation, so when training it in a vertical field, a limited number of sample pairs can be used to achieve a good training effect; The street view fusion features of a geographic region can more clearly reflect the common features of the geographic region, while the text feature vectors of the text labels of street view image samples of different geographic regions often have obvious differences. Therefore, the multimodal fusion features of multiple geographic regions formed by combining the text feature vectors of the text labels of street view image samples of multiple geographic regions with the street view fusion features of multiple geographic regions can often better highlight the uniqueness of each geographic region, which is conducive to eliminating the influence of interference factors and other contents in the sample on the recognition, and is conducive to avoiding the difficulties in the recognition of geographic regions caused by the similar image content in a small area, so as to enable the model to converge quickly during the training process, and to improve the recognition performance of the successfully trained model. In addition, by representing the geographic region with at least four gradually reduced regional ranges, it is conducive to further improving the small-scale precise positioning ability of the successfully trained model; it can be seen that the technical solution provided by the present disclosure is conducive to improving the training efficiency of the model and enriching the implementation method of geographic location recognition, while improving the accuracy of geographic location recognition.

[0101] Figure 6 FIG. 1 is a flow chart of an embodiment of a method for identifying a geographic location of a street view image disclosed in the present invention. Figure 6 The method shown mainly includes: S600, S601, S602, S603 and S604. In one example, Figure 6 Some of the steps in can be implemented using a successfully trained multimodal model based on contrastive learning, and the identified geographic locations are related to the geographic areas involved in training. Figure 6 and Figure 7 The specific steps included in the method for identifying geographic location of street view images are described.

[0102] S600: providing a street scene image to be identified as an input image to a multimodal model based on contrastive learning to obtain an image feature vector of the street scene image to be identified.

[0103] The street view image to be identified in the present disclosure generally refers to an image containing human features, for example, an image containing a building, an image containing a road, or an image containing an inscription / road sign, etc. An example of a street view image to be identified is Figure 7 Bottom left corner of the photo.

[0104] In one example, a street scene image to be identified is provided as an input image to an image encoder in a multimodal model, and the image encoder performs feature extraction processing on the street scene image to be identified and outputs an image feature vector of the street scene image to be identified.

[0105] S601: Obtain text feature vectors of text labels of street view images of multiple geographic areas.

[0106] Specifically, the geographical area in this step can refer to the description in the above embodiment, and will not be described in detail here. The present disclosure can use multiple methods to obtain the text feature vectors of the street view image text labels of multiple geographical areas, and the following two methods are exemplified:

[0107] Method 1: The present disclosure pre-sets a label sample set, which includes: street view image sample text labels of all geographical areas (such as Figure 7 (2) in the figure shows the text labels corresponding to n geographical areas, which are the street view label of area 1, the street view label of area 2, the street view label of area 3, ..., and the street view label of area n). The text labels of the street view image samples of all geographical areas in the label sample set can be provided to the text encoder in the model, so that the text feature vectors of the text labels of the street view image samples of multiple different geographical areas in the label sample set can be obtained according to the output of the text encoder. The operations performed by the text encoder include: for the text labels of the street view image samples of any geographical area in the label sample set, the text labels can be firstly segmented by using the word segmentation module, and each word after the segmentation processing can be converted into vector data respectively; then, the converted vector data can be extracted by using the text feature extraction module, so as to obtain the text feature vector of the text labels of the street view image samples of the geographical area.

[0108] Method 2: The present disclosure pre-sets a text feature vector set, which includes: text feature vectors of text labels of street view image samples of all geographical areas, and the present disclosure can directly read all text feature vectors from the set. The set is usually established during the model training process.

[0109] S602, text feature vectors of text labels of street view images of multiple geographical areas and street view fusion features of multiple geographical areas are provided to a multimodal model, and the multimodal model obtains multimodal fusion features of multiple geographical areas formed by combining the text feature vectors of text labels of street view image samples of multiple geographical areas with the street view fusion features of multiple geographical areas.

[0110] Specifically, the text feature vectors of the text labels of the street view images of multiple geographic areas and the street view fusion features of multiple geographic areas can be provided to the feature merging module in the model, and the feature merging module performs the operation of combining the text feature vectors of the text labels of the street view image samples of multiple geographic areas with the street view fusion features of multiple geographic areas respectively, so as to obtain the multimodal fusion features of multiple geographic areas according to the output of the feature merging module.

[0111] In one example, for any geographical area, the feature merging module of the present disclosure may combine the text feature vector of the text label of the street view image sample of the geographical area with the street view fusion feature of the geographical area based on the attention mechanism (e.g. Figure 7 In (3), T 1 With FF 1 Combine, T 2 With FF 2 Combine, T 3 With FF 3 Combine, ..., and T n With FF n Combined), so as to obtain the multimodal fusion features of the geographical area (such as Figure 7 T 1 '、T 2 '、T 3 ', ... and T n '). For example, for any geographical area, the feature merging module adjusts the text feature vector of the text label of the street view image sample of the geographical area according to the first weight to obtain a first feature vector; and adjusts the street view fusion feature of the geographical area according to the second weight to obtain a second feature vector; then, the feature merging module calculates the vector sum of the first feature vector and the second feature vector (such as the vector sum calculated using the above formula (3)), and outputs it, so as to obtain the multimodal fusion feature of the geographical area according to the output of the feature merging module.

[0112] It should also be noted that if during the model training process, a set of street view fusion features is finally formed (such as Figure 7 In (1), that is, the street view fusion feature library FF), when the street view image samples of each geographical area do not change, the present disclosure can also directly read the street view fusion features of each geographical area from the street view fusion feature library FF (that is, Figure 7 (1) of area 1 street view fusion features, area 2 street view fusion features, ... and area n street view fusion features), thereby avoiding the need to perform feature fusion operations on street view image samples of each geographical area each time the geographical location of the street view image to be identified is performed. Furthermore, if a multimodal fusion feature set of multiple geographical areas is finally formed during the model training process, and the text labels of the street view images of the multiple geographical areas do not change, and the street view fusion features of the multiple geographical areas do not change, then the present disclosure can also directly read the multimodal fusion features of the multiple geographical areas from the set, thereby avoiding the need to perform operations of extracting text feature vectors and combining and generating multimodal fusion features of multiple geographical areas each time the geographical location of the street view image to be identified is performed.

[0113] S603: The image feature vector and the multimodal fusion features of multiple geographical areas are provided to the multimodal model, and similarity calculation is performed via the multimodal model to obtain multiple similarities.

[0114] The similarity in the present disclosure is used to indicate the possibility that the street view image to be identified belongs to a certain geographical area, and the similarity in the present disclosure can be represented by the result of dot product calculation. Specifically, the image feature vector of the street view image to be identified and the multimodal fusion features of all geographical areas are provided as input to the similarity calculation module in the model, and the similarity calculation module performs dot product calculations on the image feature vector and the multimodal fusion features of each geographical area, thereby obtaining and outputting multiple dot product calculation results. The present disclosure obtains multiple similarities (such as Figure 7 I 1 T 1 '、I 1 T 2 '、I 1 T 3 ', ..., and I 1 T n '), one similarity corresponds to one geographical area.

[0115] S604: Determine the geographic location of the street view image to be identified according to multiple similarities.

[0116] Specifically, the present disclosure can select the maximum similarity from all similarities (such as Figure 7 The third similarity I 1 T 3' is the maximum similarity), thereby obtaining the geographical area corresponding to the maximum similarity (such as the third geographical area), and using the geographical area as the geographical location of the street view image to be identified by the multimodal model. The finally identified geographical location can be expressed as: This is a picture of a custom area (expressed in longitude and latitude) of a certain district / county / street / township in a certain city (such as Figure 7 (2) in area 3 of Street View tag).

[0117] Exemplary Devices

[0118] Figure 8 The schematic diagram of the structure of an embodiment of the model training device of the present disclosure. The model trained by the device of the embodiment is a multimodal model based on contrastive learning, which includes: an image encoder, a text encoder, a feature merging module, and a similarity calculation module; and the successfully trained multimodal model based on contrastive learning can be used to implement the embodiment of the geographic location recognition method of the street view image of the present disclosure.

[0119] like Figure 8 The device shown includes: a first module 800, a second module 801, a third module 802, a fourth module 803 and a loss module 804. The device may also optionally include: a sample division module 805, a user-defined region module 806, and a sample feature fusion encoder 807.

[0120] The first module 800 is mainly used to obtain image feature vectors of street view image samples in an image sample set of a geographical area using a multimodal model, wherein the geographical area may include: at least four area ranges, and the at least four area ranges decrease step by step. In one example, the at least four area ranges may include: city, district / county, township / street and custom area, wherein the custom area may be represented by the coordinates of a position point in a predetermined area; wherein the predetermined area may be: a range formed by taking a predetermined point in the township / street area as the center of the circle and a predetermined earth spherical distance as the radius (such as 1000 meters). In one example, the first module 800 may use an image encoder to execute the step of obtaining image feature vectors of street view image samples in an image sample set of a geographical area, and the image encoder may be an image encoder in a CLIP model.

[0121] The second module 801 is mainly used to obtain text feature vectors of text labels of street view image samples of multiple different geographical areas in the label sample set using a multimodal model. In an example, the second module 801 can use a text encoder to perform the step of obtaining text feature vectors of text labels of street view image samples of multiple different geographical areas in the label sample set, and the text encoder can include: a word segmentation module in the CLIP model and a text feature extraction module in the CLIP model. For example, the second module 801 uses the word segmentation module to perform word segmentation processing on the text labels of street view image samples of multiple different geographical areas in the label sample set, and converts each word after word segmentation processing into vector data; the second module 801 uses the text feature extraction module to extract features from the vector data to obtain text feature vectors of text labels of street view image samples of multiple different geographical areas.

[0122] The third module 802 is mainly used to combine the text feature vectors of the street view image sample text labels of multiple geographical regions with the street view fusion features of multiple geographical regions respectively using the multimodal model to obtain the multimodal fusion features of multiple geographical regions. In one example, the third module 802 can use the feature merging module to perform the step of combining the text feature vectors of the street view image sample text labels of multiple geographical regions with the street view fusion features of multiple geographical regions respectively. Specifically, for any geographical region, the feature merging module combines the text feature vectors of the street view image sample text labels of the geographical region with the street view fusion features of the geographical region based on the attention mechanism to obtain the multimodal fusion features of the geographical region. Furthermore, the feature merging module can adjust the text feature vectors of the street view image sample text labels of the geographical region according to the first weight to obtain the first feature vector; the feature merging module can adjust the street view fusion features of the geographical region according to the second weight to obtain the second feature vector; the feature merging module can calculate the vector sum of the first feature vector and the second feature vector to obtain the multimodal fusion features of the geographical region.

[0123] The fourth module 803 is mainly used to use the multimodal model to obtain the similarity between the image feature vector and the multimodal fusion features of multiple geographical regions. In an example, the fourth module 803 can use the similarity calculation module to perform the step of obtaining the similarity between the image feature vector and the multimodal fusion features of multiple geographical regions.

[0124] The loss module 804 is mainly used to calculate the loss based on the similarity and the geographic area label of the street view image sample, and adjust the parameters of the multimodal model according to the result of the loss calculation. For example, the loss module 804 adjusts the parameters of the text encoder and the image encoder in the multimodal model according to the result of the loss calculation.

[0125] The sample division module 805 is mainly used to calculate the earth's spherical distances of all street view image samples and the predetermined points according to the coordinates of all street view image samples and the coordinates of the predetermined points, and to determine the image sample sets of the geographical areas to which all street view image samples belong according to the earth's spherical distances and the radius of each predetermined area.

[0126] The custom area module 806 is mainly used to calculate the mean value of the coordinates of all street view image samples in the image sample set for any geographical area, and use the result of the mean value calculation as the coordinates for describing the custom area in the geographical area.

[0127] The sample feature fusion encoder 807 is mainly used to perform feature fusion processing on the street view image samples in the image sample set of each geographical area by using the sample feature fusion encoder to obtain the street view fusion features of each geographical area. In one example, the sample feature fusion encoder may include: an input layer, a deep convolutional neural network based on a residual network architecture with the classification layer discarded, and a mean calculation module; specifically, for any geographical area, multiple street view image samples in the image sample set of the geographical area are respectively provided to the input layer as input images, and the input layer is mainly used to convert the multiple street view image samples into vector data respectively; the vector data of the multiple street view image samples are respectively provided to the deep convolutional neural network, and the deep convolutional neural network is mainly used to obtain the image feature vectors of the multiple street view image samples; the mean calculation module is mainly used to calculate the mean of each vector element in the image feature vectors of the multiple street view image samples to obtain the street view fusion features of the geographical area.

[0128] Fig. 9 The schematic diagram is a schematic diagram of an embodiment of a device for identifying geographic location of street view images disclosed in the present invention. The device of this embodiment uses the successfully trained multimodal model based on contrastive learning to realize geographic location identification of street view images.

[0129] like Fig. 9 The device shown includes: a fifth module 900 , a sixth module 901 , a seventh module 902 , an eighth module 903 and an identification module 904 .

[0130] The fifth module 900 is mainly used to provide the street scene image to be identified as an input image to the successfully trained multimodal model based on contrastive learning, so as to obtain the image feature vector of the street scene image to be identified.

[0131] The sixth module 901 is mainly used to obtain text feature vectors of street view image text labels of multiple geographical areas, wherein the geographical areas include at least four area ranges, and the at least four area ranges decrease in a step-by-step manner.

[0132] The seventh module 902 is mainly used to provide the text feature vectors of the text labels of the street view images of multiple geographical areas and the street view fusion features of multiple geographical areas to the multimodal model, and obtain the multimodal fusion features of multiple geographical areas by combining the text feature vectors of the text labels of the street view image samples of multiple geographical areas with the street view fusion features of multiple geographical areas through the multimodal model.

[0133] The eighth module 903 is mainly used to provide the image feature vector and the multimodal fusion features of multiple geographical areas to the multimodal model, and perform similarity calculation via the multimodal model to obtain multiple similarities.

[0134] The identification module 904 is mainly used to determine the geographical location of the street view image to be identified according to multiple similarities.

[0135] Exemplary Electronic Devices

[0136] Reference below Fig.10 An electronic device according to an embodiment of the present disclosure is described. Fig.10 1 shows a block diagram of an electronic device according to an embodiment of the present disclosure. Fig.10 As shown, the electronic device 1001 includes one or more processors 1011 and a memory 1012 .

[0137] The processor 1011 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 1001 to perform desired functions.

[0138] The memory 1012 may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory, for example, may include: random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory, for example, may include: read-only memory (ROM), hard disk, and flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 1011 may execute the program instructions to implement the geographic location recognition method of the street view image, the model training method, and / or other desired functions of the various embodiments of the present disclosure described above.

[0139] In one example, the electronic device 1001 may also include: an input device 1013 and an output device 1014, etc., which are interconnected through a bus system and / or other forms of connection mechanisms (not shown). In addition, the input device 1013 may also include, for example, a keyboard, a mouse, etc. The output device 1014 may output various information to the outside. The output device 1014 may include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, etc.

[0140] Of course, to simplify, Fig.10 Only some of the components related to the present disclosure in the electronic device 1001 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device 1001 may also include any other appropriate components.

[0141] Exemplary computer program products and computer-readable storage media

[0142] In addition to the above-mentioned methods and devices, an embodiment of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the street view image geographic location recognition method and the model training method according to various embodiments of the present disclosure described in the above-mentioned "Exemplary Method" section of this specification.

[0143] The computer program product may be written in any combination of one or more programming languages ​​to write program code for performing the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0144] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, causes the processor to execute the steps of the street view image geographic location recognition method and the model training method according to various embodiments of the present disclosure described in the above “Exemplary Method” section of this specification.

[0145] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive enumeration) of readable storage media can include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0146] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. must be possessed by each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, rather than limitation, and the above details do not limit the present disclosure to the necessity of adopting the above specific details to be implemented.

[0147] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0148] The block diagrams of the devices, apparatuses, equipment, and systems involved in the present disclosure are intended only as illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems may be connected, arranged, and configured in any manner. Words such as "include," "comprise," "have," and the like are open words, meaning "including but not limited to," and may be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or," and may be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and may be used interchangeably therewith.

[0149] The method and apparatus of the present disclosure may be implemented in many ways. For example, the method and apparatus of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above, unless otherwise specifically stated. In addition, in some embodiments, the present disclosure may also be implemented as a program recorded in a recording medium, which includes machine-readable instructions for implementing the method according to the present disclosure. Therefore, the present disclosure also covers a recording medium storing a program for executing the method according to the present disclosure.

[0150] It should also be noted that in the apparatus, device and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0151] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects, etc., will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0152] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A model training method, characterized in that: The training method is used to train a multimodal model based on contrastive learning, and the model training method includes: Using the multimodal model, an image feature vector of a street view image sample in an image sample set of a geographical area is obtained, wherein the geographical area includes: at least four area ranges, and the at least four area ranges decrease in a step-by-step manner; Using the multimodal model to obtain text feature vectors of text labels of street view image samples of multiple different geographical areas in the label sample set; Using the multimodal model, the text feature vectors of the text labels of the street view image samples of the multiple geographical areas are respectively combined with the street view fusion features of the multiple geographical areas to obtain the multimodal fusion features of the multiple geographical areas; Using the multimodal model to obtain similarities between the image feature vector and the multimodal fusion features of the multiple geographic areas; Performing loss calculation according to the similarity and the geographic area label of the street view image sample, and adjusting the parameters of the multimodal model according to the result of the loss calculation; Among them, the successfully trained multimodal model based on contrastive learning is used to determine the geographic location of the street view image to be identified; The at least four regions include: city, district / county, township / street and custom region; The custom area is represented by the coordinates of a location point within the predetermined area, and the predetermined area includes: a range formed by taking a predetermined point within the township / street area as the center and a predetermined earth spherical distance as the radius.

2. The model training method according to claim 1, characterized in that: The method further comprises: Calculating the distances of all street view image samples from the predetermined point on the earth's spherical surface according to the coordinates of all street view image samples and the coordinates of the predetermined point; According to the earth's spherical distance and the radius of each predetermined area, an image sample set of the geographical area to which all the street view image samples belong is determined.

3. The model training method according to claim 2, characterized in that: The method further comprises: For an image sample set of any geographic area, the coordinates of all street view image samples in the image sample set are averaged, and the result of the average calculation is used as the coordinates for describing the custom area in the geographic area.

4. The model training method according to any one of claims 1 to 3, characterized in that: The multimodal model includes: an image encoder, a text encoder, a feature merging module, and a similarity calculation module; The image encoder is used to perform the step of obtaining the image feature vector of the street view image sample in the image sample set of a geographical area, and the image encoder includes: an image encoder based on a pre-trained CLIP model for text-image pairs; The text encoder is used to execute the step of obtaining text feature vectors of text labels of street view image samples of multiple different geographical areas in the label sample set, and the text encoder includes: a word segmentation module in the CLIP model and a text feature extraction module in the CLIP model; The feature merging module is used to execute the step of combining the text feature vectors of the text labels of the street view image samples of the multiple geographical areas with the street view fusion features of the multiple geographical areas respectively to obtain the multimodal fusion features of the multiple geographical areas; The similarity calculation module is used to execute the step of obtaining the similarity between the image feature vector and the multimodal fusion features of the multiple geographical areas respectively; The step of obtaining text feature vectors of text labels of street view image samples of multiple different geographical areas in the label sample set includes: Using the word segmentation module to perform word segmentation processing on the text labels of street view image samples of multiple different geographical areas in the label sample set, and converting each word after the word segmentation processing into vector data; The text feature extraction module is used to perform feature extraction on the vector data to obtain text feature vectors of the text labels of the street view image samples of the multiple different geographical areas.

5. The model training method according to claim 4, characterized in that: The multimodal model further includes: a sample feature fusion encoder; and the sample feature fusion encoder includes: an input layer, a deep convolutional neural network based on a residual network architecture with a classification layer discarded, and a mean calculation module; The method further comprises: For any geographical area, a plurality of street view image samples in the image sample set of the geographical area are respectively provided to the input layer as input images, and the input layer converts the plurality of street view image samples into vector data respectively; The vector data of the plurality of street view image samples are respectively provided to the deep convolutional neural network, and image feature vectors of the plurality of street view image samples are obtained via the deep convolutional neural network; The mean value calculation module calculates the mean value of each vector element in the image feature vector of the plurality of street view image samples to obtain the street view fusion feature of the geographical area.

6. The model training method according to claim 4, characterized in that: The step of combining the text feature vectors of the text labels of the street view image samples of the multiple geographical areas with the street view fusion features of the multiple geographical areas to obtain the multimodal fusion features of the multiple geographical areas includes: For any geographical area, the feature merging module combines the text feature vector of the text label of the street view image samples of the geographical area with the street view fusion feature of the geographical area based on the attention mechanism to obtain the multimodal fusion feature of the geographical area.

7. A method for identifying geographic location of a street view image, characterized in that: The method comprises: Providing the street scene image to be identified as an input image to the multimodal model based on contrastive learning to obtain an image feature vector of the street scene image to be identified; Acquire text feature vectors of text labels of street view images of multiple geographical areas, wherein the geographical areas include: at least four area ranges, and the at least four area ranges decrease in a step-by-step manner; The text feature vectors of the street view image text labels of the multiple geographical areas and the street view fusion features of the multiple geographical areas are provided to the multimodal model, and the multimodal fusion features of the multiple geographical areas are obtained by combining the text feature vectors of the street view image sample text labels of the multiple geographical areas with the street view fusion features of the multiple geographical areas through the multimodal model; The image feature vector and the multimodal fusion features of the multiple geographical areas are provided to the multimodal model, and similarity calculation is performed by the multimodal model to obtain multiple similarities; Determining the geographic location of the street view image to be identified according to the multiple similarities; The at least four regions include: city, district / county, township / street and custom region; The custom area is represented by the coordinates of a location point within the predetermined area, and the predetermined area includes: a range formed by taking a predetermined point within the township / street area as the center and a predetermined earth spherical distance as the radius.

8. A model training device, characterized in that: The training device is used to train a multimodal model based on contrastive learning, and the model training device includes: A first module is used to obtain an image feature vector of a street view image sample in an image sample set of a geographical area by using the multimodal model, wherein the geographical area includes at least four area ranges, and the at least four area ranges decrease in a step-by-step manner; The second module is used to obtain text feature vectors of text labels of street view image samples of multiple different geographical areas in the label sample set using the multimodal model; A third module is used to combine the text feature vectors of the text labels of the street view image samples of the multiple geographical areas with the street view fusion features of the multiple geographical areas respectively by using the multimodal model to obtain the multimodal fusion features of the multiple geographical areas; A fourth module is used to obtain similarities between the image feature vector and the multimodal fusion features of the multiple geographical areas respectively using the multimodal model; A loss module, configured to perform loss calculation according to the similarity and the geographic area label of the street view image sample, and adjust the parameters of the multimodal model according to the result of the loss calculation; Among them, the successfully trained multimodal model based on contrastive learning is used to determine the geographic location of the street view image to be identified; The at least four regions include: city, district / county, township / street and custom region; The custom area is represented by the coordinates of a location point within the predetermined area, and the predetermined area includes: a range formed by taking a predetermined point within the township / street area as the center and a predetermined earth spherical distance as the radius.

9. A device for identifying geographic location of a street view image, characterized in that: The identification device comprises: A fifth module is used to provide the street scene image to be identified as an input image to the multimodal model based on contrastive learning to obtain an image feature vector of the street scene image to be identified; A sixth module is used to obtain text feature vectors of street view image text labels of multiple geographical areas, wherein the geographical areas include: at least four area ranges, and the at least four area ranges decrease in a step-by-step manner; A seventh module is used to provide the text feature vectors of the street view image text labels of the multiple geographical areas and the street view fusion features of the multiple geographical areas to the multimodal model, and obtain, through the multimodal model, the multimodal fusion features of the multiple geographical areas formed by combining the text feature vectors of the street view image sample text labels of the multiple geographical areas with the street view fusion features of the multiple geographical areas; An eighth module is used to provide the image feature vector and the multimodal fusion features of the multiple geographical areas to the multimodal model, and perform similarity calculation via the multimodal model to obtain multiple similarities; an identification module, configured to determine the geographic location of the street view image to be identified according to the multiple similarities; The at least four regions include: city, district / county, township / street and custom region; The custom area is represented by the coordinates of a location point within the predetermined area, and the predetermined area includes: a range formed by taking a predetermined point within the township / street area as the center and a predetermined earth spherical distance as the radius.

Images