A localization optimization method based on multimodal data

Abstract

This invention discloses a positioning optimization method based on multimodal data, belonging to the fields of geographic information processing and social media data analysis. It first acquires user address text, images, and check-in location information from social media. After segmenting the address text into words, it classifies them into point, line, and polygon categories and constructs a candidate set. Multimodal coordinates are integrated to form a pre-selected point candidate set. The final point candidate set is obtained through spatial constraint cyclic filtering. After weight calculation and Gaussian kernel function modeling, the coordinates with the highest probability are output as the user's final location, effectively improving positioning accuracy and reliability.

Description

Technical Field

[0001] This invention belongs to the field of geographic information processing and social media data analysis technology, and specifically relates to a positioning optimization method based on multimodal data. Background Technology

[0002] With the rapid development of social media, online platforms have accumulated massive amounts of location information data containing multimodal features such as text, images, and coordinates. This data not only fully records users' daily behavioral trajectories but also provides new data support for multiple fields such as user positioning, emergency rescue, and urban management. Especially on social platforms such as Weibo, the text content posted by users, the images they upload, and their check-in records all contain rich geographic information, becoming important clues to characterize users' geographical locations.

[0003] Most current mainstream social media data location technologies rely on a single information modality for location analysis. For example, they may rely solely on a user's location check-in records or address text for analysis. When this crucial information is missing, the system often struggles to produce effective location results. Furthermore, the significant uncertainty in the accuracy of user location data further exacerbates the inconsistency in the reliability and precision of location results. For instance, in address text location analysis, if the address text contains detailed place name information, the text parsing location accuracy can reach the house level; however, if the address text is vague or only mentions directional words, the location error will increase significantly. Similarly, in image location analysis, the effectiveness highly depends on the completeness of the image database and the detail of the input image information—if the features in the image are highly recognizable (such as the Forbidden City), and there are matching samples for that location in the image database, high-precision location can be achieved; conversely, if the image information is incomplete or the features are unclear, the location accuracy will drop significantly.

[0004] In summary, existing positioning methods have significant shortcomings: most studies rely on single-modal data and fail to fully leverage the complementary advantages of multimodal data, resulting in poor positioning performance. Therefore, this paper proposes a positioning optimization method based on multimodal data. This method comprehensively utilizes multimodal information such as text, images, and check-in locations to improve positioning performance. Summary of the Invention

[0005] This invention proposes a positioning optimization method based on multimodal data, which can comprehensively utilize multimodal information such as text, images, and check-in locations to improve positioning performance.

[0006] To achieve the above objectives, the present invention proposes the following technical content:

[0007] A localization optimization method based on multimodal data includes the following steps:

[0008] S1: Obtain all address texts of users in social media software within a certain time period, segment them into words, determine the address type of each address text, and classify them into the corresponding candidate set;

[0009] If the last segment of the address text after word segmentation is a point address, then the address text is converted into coordinates through the geocoding platform and included in the "text point candidate set";

[0010] If the last segment of the address text after word segmentation is a line address, the corresponding feature vector is obtained through vector matching, and the feature vector is included in the "text line candidate set";

[0011] If the last segment of the address text after word segmentation is a face address, the corresponding feature vector is obtained through vector matching, and the feature vector is included in the "text face candidate set";

[0012] S2: Input all the image location information of the user in social media software during the same time period as in S1 into the open interface of the network image encoding platform. The network image encoding platform will convert the corresponding image location information into corresponding coordinates and include them in the "image point candidate set".

[0013] S3: Obtain the coordinates of all check-in location information of users in social media software during the same time period as in S1 by direct access, and include them in the "check-in location candidate set";

[0014] S4: Construct a preliminary candidate set of points based on the candidate sets of text points, image points, and check-in locations; set a spatial constraint circle, and filter the "coordinates" in the preliminary candidate set of points based on the spatial constraint circle, removing coordinates that do not belong to the spatial constraint circle to form the final candidate set of points;

[0015] S5: Count the number of coordinates in the final candidate point set from the text point candidate set, image point candidate set, and check-in location point candidate set, respectively, and set them as follows: n 1. n 2 and n 3; Set the number of coordinates in the candidate set of final points to be n , n = n 1+ n 2+ n 3. Based on whether the candidate set of text lines is empty, obtain the distance parameter of each coordinate, and calculate its distance decay term based on the distance parameter of each coordinate; based on the distance decay term of each coordinate, calculate the normalized weight of each coordinate.

[0016] S6: Based on the normalized weights of each coordinate and combined with the Gaussian kernel function, calculate the Gaussian kernel surface prediction probability for each coordinate in the spatial constraint circle; take the coordinate represented by the maximum Gaussian kernel surface prediction probability as the final user location.

[0017] Further, step S4 includes the following steps:

[0018] S4.1: Construct a candidate set of pre-selected points, using the following formula:

[0019]

[0020] In the formula, This represents the union operation; Represents the candidate set of text points; This represents the candidate set of image points; This represents the candidate set of check-in locations;

[0021] S4.2: Filter the coordinates of the candidate points in the pre-selection set; specifically including the following steps:

[0022] S4.2.1: If the text line candidate set Candidate set of text All are empty sets, so the candidate set of pre-selected points will be... The bounding rectangle formed by all coordinates in the bounding rectangle serves as the spatial constraint circle. S No need to remove the candidate set of pre-selected points The data in the sample retains the candidate set of pre-selected points. Data in the middle;

[0023] S4.2.2: If the text line candidate set Empty set, candidate set for text face If the set is non-empty, then establish a spatial constraint cycle:

[0024]

[0025] If for the candidate set of preselected points The Middle i coordinates ( x i , y i ) ，i ∈[1, m ], and is an integer; m Represents the candidate set of preselected points The total number of coordinates in; if If established, then the first one is retained. i coordinates ( x i , y i );like If this is not true, then delete the first one.i coordinates ( x i , y i );

[0026] S4.2.3: If the text line candidate set Non-empty set, candidate set of text face If the set is empty, then the spatial constraint cyclic structure is:

[0027]

[0028] In the formula, dist (*,*) represent the corresponding coordinates. x,y ) to text line candidate set The shortest distance to a line-like vector in the vector; d b This indicates the set buffer distance;

[0029] For the candidate set of preselected points The Middle i coordinates ( x i , y i ) ，i ∈[1, m ], and is an integer; if If established, then the first one is retained. i coordinates ( x i , y i );like If this is not true, then delete the first one. i coordinates ( x i , y i );

[0030] S4.2.4: If the text line candidate set Candidate set of text If none of them are empty sets, then the spatial constraint cyclic set is defined as follows:

[0031]

[0032] For the candidate set of preselected points The Middle i coordinates ( x i , y i ) ，i ∈[1, m ], and is an integer; if or If the coordinate is true, retain it; otherwise, delete it, forming the final candidate set of points. .

[0033] Furthermore, step S5 includes the following steps:

[0034] S5.1: Obtain the distance parameter for each coordinate;

[0035] The first method: when the text line candidate set When it is an empty set, the first k Distance parameters of each coordinate d k Set value;

[0036] The second method: when the text line candidate set When it is a non-empty set, the first... k Coordinates and candidate sets of text lines The minimum shortest distance among all line vectors is used as the first... k Distance parameters of each coordinate d k ;

[0037] S5.2: Calculate the distance decay term for each coordinate; the formula is:

[0038]

[0039] In the formula, d b This indicates the set buffer distance; d k express k Distance parameters for each coordinate; s k Indicates the first k Distance decay term for each coordinate;

[0040] S5.3: Calculate the basic weight for each coordinate; based on the basic weight, calculate the weight of each coordinate before normalization; specifically including the following steps:

[0041] S5.3.1: Based on the candidate point set from which each coordinate originates, and in conjunction with the set weights of the corresponding candidate point set, calculate the basic weight of that coordinate; the formula is:

[0042]

[0043] In the formula, Indicates the candidate set of the final point The mid-coordinates are derived from the candidate set of text points. The basic weight of time; Indicates the candidate set of the final point The midpoint coordinates are derived from the candidate set of image points. The basic weight of time; Indicates the candidate set of the final point The mid-coordinate is derived from the candidate set of check-in locations. The basic weight of time; M 1. M 2 and M 3 represent the candidate sets of text points respectively. Image candidate set and check-in location candidate set The weighting of the settings; n 1. n 2 and n 3 indicates the final point candidate set The data in the middle are derived from the candidate set of text points. Image candidate set and check-in location candidate set The number of coordinates; if n 1. n 2 or n If 3 is 0, then it does not have the corresponding basic weight;

[0044] S5.3.2: Obtain the final point candidate set through the operations in S5.3.1. The Middle k The basic weight of each coordinate is set as follows: ;

[0045] S5.4: Calculate the first k The weights of each coordinate before normalization; the formula is:

[0046]

[0047] In the formula, express k The basic weight of each coordinate; Indicates the first k Weights of each coordinate before normalization; s k Indicates the first k Distance decay term for each coordinate;

[0048] S5.5: Calculate the first k The normalized weights of each coordinate; the formula is:

[0049]

[0050] In the formula, Indicates the first k Weights after coordinate normalization; Indicates the first j The weights of each coordinate before normalization; x k ,y k ) represents the candidate set of the final point. The Middle k Coordinates.

[0051] Further, step S6 includes the following steps:

[0052] S6.1: Computational Space Constraints S Any point in the middle and the candidate set of the final point The predicted probability of the Gaussian kernel surface for all coordinates is given by the formula:

[0053]

[0054] In the formula, Representing spatial constraint loops S any point in it; Representing spatial constraint loops S Any point in the candidate set of the final point Predicted probabilities of Gaussian kernel surfaces at all coordinates; K (*) represents the Gaussian kernel function; h This represents the Gaussian kernel bandwidth; where the Gaussian kernel function is expressed as: In the formula, r Represents Gaussian kernel variables;

[0055] S6.2: The coordinates represented by the maximum predicted probability of the Gaussian kernel surface are used as the end-user location; the formula is:

[0056]

[0057] In the formula, Indicates the end-user location; This indicates taking the coordinate with the highest probability.

[0058] The beneficial effects that can be achieved by adopting the above technologies are:

[0059] 1. Fully integrate multimodal location information from social media text, images, and check-ins; filter out invalid data through address semantic classification and spatial constraints to solve the problem of missing or insufficient accuracy of single-modal location information and improve positioning reliability;

[0060] 2. By introducing a distance attenuation term and Gaussian kernel probability modeling, the positioning value of multimodal data can be accurately mined, significantly improving the accuracy of user location prediction. Attached Figure Description

[0061] Figure 1 This is the logic diagram of this solution;

[0062] Figure 2 This is a schematic diagram of the predicted probability of the Gaussian kernel surface;

[0063] Figure 3 It represents the positioning accuracy of each method in the example;

[0064] Figure 4 It is the average error of each method in the example. Detailed Implementation

[0065] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0066] like Figure 1 As shown, a localization optimization method based on multimodal data specifically includes the following steps:

[0067] S1: Obtain all address texts of users in social media software within a certain time period, segment them into words, determine the address type of each address text, and classify them into the corresponding candidate set.

[0068] Within a time period T, all address texts from users' social media apps are segmented using a bidirectional long short-term memory network and a conditional random field. The address type of the address text is determined based on the last segment and then assigned to the corresponding candidate set.

[0069] Specifically:

[0070] If the last word of an address text is a point-type address such as "number", "building", or "square", then the address text is converted into coordinates through a geocoding platform and included in the "text point candidate set". middle;

[0071] If the last word of an address text is a line-type address such as "road," "path," "street," "river," or "canal," the corresponding feature vector is obtained through vector matching, and this feature vector is directly added to the "text line candidate set," i.e., it is included in the list of candidate addresses. middle;

[0072] If the last word of an address text is a facet address such as "district", "county", "city", "province", or "district", the corresponding feature vector is obtained through vector matching, and this feature vector is directly added to the "text facet candidate set". middle;

[0073] For example:

[0074] If an address text is: No. xxx, Tianyin Avenue, Jiangning District, Nanjing City, Jiangsu Province, and the segmented words are: "Jiangsu Province", "Nanjing City", "Jiangning District", "Tianyin Avenue", "No. xxx", with the last word being "No.", then the address text is first converted into coordinates using a geocoding platform, and then the coordinates are assigned to... middle;

[0075] If the address type of the last segment cannot be determined, the random forest method is used to judge the semantics and word vector similarity of the segment to finally identify its address type. For example, in the sentence "Jiangsu Province, Nanjing City, Jiangning District, Wending Guangchang", the segmented words are "Jiangsu Province", "Nanjing City", "Jiangning District", and "Wending Guangchang". After the random forest method judges the last segment "Wending Guangchang", it is determined that there is a text error. It should actually be "Wending Plaza". Therefore, this address text belongs to the "point-type address".

[0076] S2: Input the location information of all images taken by the user in social media software during the same time period as in S1 into the open interface of a network image encoding platform (such as GeoSpy). The network image encoding platform will convert the corresponding image location information into corresponding coordinates and add them to the "image point candidate set". middle;

[0077] S3: For all user check-in location information within the same time period as in S1, obtain the coordinates of the check-in location information directly and add them to the "check-in location candidate set". middle;

[0078] S4: Candidate Set Based on Text Points Image candidate set and check-in location candidate set Constructing a candidate set of pre-selected points Define a spatial constraint circle, and then select candidate points based on the spatial constraint circle. The coordinates in the data are filtered to remove coordinates that do not belong to the spatial constraint circle, forming the final candidate set of points. Specifically, it includes the following steps:

[0079] S4.1: Constructing a candidate set of pre-selected points The formula is:

[0080]

[0081] In the formula, This indicates the union operation.

[0082] S4.2: Candidate set of pre-selected points The data in the dataset is then filtered. This includes the following steps:

[0083] S4.2.1: If the text line candidate set Candidate set of text All are empty sets, so the candidate set of pre-selected points will be... The bounding rectangle formed by all coordinates in the bounding rectangle serves as the spatial constraint circle. S No need to remove the candidate set of pre-selected points The data in the sample retains the candidate set of pre-selected points. All data in;

[0084] S4.2.2: If the text line candidate set Empty set, candidate set for text face If the set is non-empty, then the spatial constraint cycle is:

[0085]

[0086] In the formula, S Represents a spatial constraint circle; Represents the candidate set of faces.

[0087] Set the candidate set of preselected points The CCP m The coordinates for the first coordinate are... i coordinates ( x i , y i ), i ∈[1, m ], and is an integer; if it is in the candidate set of the face. In the middle (i.e., located in the spatial constraint circle) S If the coordinate is in the middle, then keep that coordinate; otherwise, delete that coordinate. That is: if... If established, then the first one is retained. i coordinates ( x i , y i );like If this is not true, then delete the first one. i coordinates ( x i , y i );

[0088] S4.2.3: If the text line candidate set Non-empty set, candidate set of text face If the set is empty, then the spatial constraint cyclic structure is: In the formula, dist (*,*) represent the corresponding coordinates. x,y ) to text line candidate set The shortest distance to a line-like vector in the vector;d b This indicates the set buffer distance, such as 1000m.

[0089] Set the candidate set of preselected points The CCP m The coordinates for the first coordinate are... i coordinates ( x i , y i ), i ∈[1, m ], and is an integer; if it reaches the candidate set of text lines The shortest distance of a certain line vector is less than or equal to the set buffer distance. d b (that is, the first) i coordinates ( x i , y i Located within the spatial constraint circle S If the coordinate is true (in the middle), then keep the coordinate; otherwise, delete the coordinate. That is: if... If established, then the first one is retained. i coordinates ( x i , y i );like If this is not true, then delete the first one. i coordinates ( x i , y i );

[0090] S4.2.4: If the text line candidate set Candidate set of text If none of them are empty sets, then the spatial constraint cyclic set is defined as follows:

[0091]

[0092] Set the candidate set of preselected points The CCP m The coordinates for the first coordinate are... i coordinates ( x i , y i ), i ∈[1, m ], and is an integer; if it reaches the candidate set of text lines The shortest distance of a certain line vector is less than or equal to the set buffer distance. d b Or if it is in the candidate set If a coordinate holds true in a certain face vector, then that coordinate is retained; otherwise, it is deleted. That is: if... or If the condition is met, the coordinate is retained; otherwise, the coordinate is deleted.

[0093] Candidate set of preselected points After the coordinates in the data are reduced, a final candidate set of points is formed. See Figure 2 .

[0094] S5: Statistical Final Point Candidate Set The two sources are respectively from the text point candidate set. Image candidate set and check-in location candidate set The number of coordinates is set to 1. n 1. n 2 and n 3; Define the candidate set of final points The number of coordinates in n , n = n 1+ n 2+ n 3; Based on the candidate set of text lines To determine if the set is empty, obtain the distance parameter for each coordinate, and calculate the distance decay term based on the distance parameter. Then, based on the distance decay term, calculate the normalized weight for each coordinate. Specifically, this includes the following steps:

[0095] S5.1: Obtain the distance parameter for each coordinate; the distance parameter can be obtained in the following two ways:

[0096] The first method: when the text line candidate set When it is an empty set, the first k Distance parameters of each coordinate d k For setting value, k ∈[1, n ], and must be an integer; for example: d k =10.

[0097] The second method: when the text line candidate set When it is a non-empty set, the first... k Coordinates and candidate sets of text lines The minimum shortest distance among all line vectors is used as the first... k Distance parameters of each coordinate d k Since each line vector is a line segment composed of several coordinate nodes, when the first line vector is known... kAfter obtaining the coordinates, the corresponding line vector coordinates can be combined to calculate the first coordinate. k If there are C line vectors, there will be C shortest distances between each coordinate and the line vector. The minimum value among these C shortest distances is selected as the distance parameter. d k .

[0098] S5.2: Calculate the distance decay term for each coordinate. The formula is:

[0099]

[0100] In the formula, d b This indicates the set buffer distance. s k Indicates the first k Distance decay term for each coordinate. d k express k Distance parameters for each coordinate.

[0101] S5.3: Calculate the basic weight for each coordinate; based on the basic weight, calculate the weight of each coordinate before normalization. This includes the following steps:

[0102] S5.3.1: Calculate the basic weight of each coordinate based on the candidate set of points from which it originates, combined with the set weight of the corresponding candidate set of points.

[0103] Set text point candidate set Image candidate set and check-in location candidate set The weights are set as follows: M 1. M 2 and M 3; Combine the candidate set of the final point Number of candidate sets for each point n 1. n 2 and n 3. Calculate the basic weights of each coordinate in the candidate set of corresponding points; the formula is:

[0104]

[0105] In the formula, Indicates the candidate set of the final point The mid-coordinates are derived from the candidate set of text points. The basic weight of time; Indicates the candidate set of the final point The midpoint coordinates are derived from the candidate set of image points. The basic weight of time; Indicates the candidate set of the final point The mid-coordinate is derived from the candidate set of check-in locations. The basic weight of time; if n 1. n 2 or n If 3 is 0, then it does not have the corresponding basic weight, such as: n If 1 is 0, then there is no candidate set derived from the text points. The basic weight of time;

[0106] S5.3.2: Obtain the final point candidate set through the operations in S5.3.1. The Middle k The basic weight of each coordinate is set as follows: ;

[0107] S5.4: Calculate the first k The weights of each coordinate before normalization. The formula is:

[0108]

[0109] In equation (2), express k The basic weight of each coordinate; Indicates the first k Weights of each coordinate before normalization; s k Indicates the first k Distance decay term for each coordinate.

[0110] S5.5: Calculate the first k The normalized weights of each coordinate. The formula is:

[0111]

[0112] In the formula, Indicates the first k Weights after coordinate normalization; Indicates the first j The weights of each coordinate before normalization; n This indicates the number of coordinates in the final point candidate set.

[0113] S6: Based on the normalized weights of each coordinate and combined with the Gaussian kernel function, calculate the Gaussian kernel surface prediction probability for each coordinate in the spatial constraint circle; use the coordinate represented by the maximum Gaussian kernel surface prediction probability as the final user location. Specifically, this includes the following steps:

[0114] S6.1: Computational Space Constraints S Any point in the middle and the candidate set of the final point The predicted probability of the Gaussian kernel surface for all coordinates in the equation. The formula is:

[0115]

[0116] In equation (4), Representing spatial constraint loops S any point in it; Representing spatial constraint loops S Any point in the candidate set of the final point Predicted probabilities of Gaussian kernel surfaces at all coordinates; K (*) represents the Gaussian kernel function; h Indicates the Gaussian kernel bandwidth; x k , y k ) represents the candidate set of the final point. The Middle k There are coordinates. The Gaussian kernel function is expressed as: In the formula, r This represents the Gaussian kernel variable.

[0117] S6.2: The coordinates represented by the maximum predicted probability on the Gaussian kernel surface are used as the end-user location. The formula is:

[0118]

[0119] In the formula, Indicates the end-user location; This indicates taking the coordinate with the highest probability.

[0120] Experimental verification: Comparison of recognition success rate and average error of different positioning methods

[0121] To verify the effectiveness of the multimodal fusion localization method proposed in this invention, a total of 335 Weibo samples were selected as the experimental dataset.

[0122] These Weibo posts all originated from the public platform Sina Weibo. All samples contained at least one of the following three types of information: a clear location of the check-in, image content that can be used for visual positioning, or text text with parsable place names; some samples contained two or three types of information simultaneously. The actual geographical coordinates of the poster were determined manually for error calculation.

[0123] Based on the aforementioned Weibo data with verifiable real locations, the recognition success rate and positioning error of four positioning methods were compared. The four methods are: positioning using check-in; positioning using text; positioning using images; and positioning using the multimodal fusion method of this solution.

[0124] in, Figure 3 The recognition success rates of four methods on all samples are given.

[0125] Experimental results show that:

[0126] The success rate of location tracking for check-ins was 91.6%;

[0127] The success rate of text localization was 79.7%;

[0128] The success rate of image location was 80.9%;

[0129] The localization success rate of the multimodal fusion method in this solution is 100%.

[0130] The results show that relying on a single location method (check-in, text, or image location) is insufficient to identify all samples. However, our proposed method maintains a high recognition success rate for all samples, significantly higher than traditional methods that rely on single pieces of information. This indicates that our proposed method integrates multiple pieces of information for comprehensive evaluation, resulting in better location performance.

[0131] For successfully identified data, the coordinates after successful localization using a single method, the coordinates after successful localization using this solution, and the coordinates as determined by a human interpreter will be compared to obtain the average error. The average error results are shown below. Figure 4 The results show:

[0132] The average error of the check-in positioning is approximately 5.1 km;

[0133] The average error of text localization is approximately 100.8 km;

[0134] The average error of the image localization method is approximately 143.0 km;

[0135] The average error of the multimodal fusion method in this scheme is approximately 1.3 km.

[0136] The results show that the proposed method has a smaller average error, is more accurate, and has greater application potential compared to positioning coordinates obtained using single information.

[0137] Based on the above-described preferred embodiments of the present invention, and through the foregoing description, those skilled in the art can make various changes and modifications without departing from the inventive concept. The technical scope of this invention is not limited to the contents of the specification, but must be determined according to the scope of the claims.

Claims

1. A positioning optimization method based on multi-modal data, characterized in that, Comprise the following steps: S1: Obtain all address texts of a user in a social media software in a period of time, and divide words, and judge the address type of each address text, and enter into the corresponding candidate set; If the last word segment of the address text after word segmentation is a point address, the address text is converted into coordinates through a geographic coding platform, and is entered into the "text point candidate set"; If the last word segment of the address text after word segmentation is a line address, the corresponding geographic vector is obtained through vector matching, and the geographic vector is entered into the "text line candidate set"; If the last word segment of the address text after word segmentation is a surface address, the corresponding geographic vector is obtained through vector matching, and the geographic vector is entered into the "text surface candidate set"; S2: Input all picture positioning information of a user in a social media software in the same period as S1 into the network picture coding platform open interface, and convert the corresponding picture positioning information into corresponding coordinates by the network picture coding platform, and enter into the "picture point candidate set"; S3: Obtain the coordinates of the check-in positioning information through direct calling, and enter into the "check-in location point candidate set"; S4: Construct a preselected point candidate set based on the text point candidate set, the picture point candidate set and the check-in location point candidate set; set a spatial constraint ring, and screen the "coordinates" in the preselected point candidate set based on the spatial constraint ring, and remove the coordinates not belonging to the spatial constraint ring to form a final point candidate set; S5: count the number of coordinates in the final point candidate set respectively from the text point candidate set, the picture point candidate set and the check-in location point candidate set, and set them as n 1, n 2 and n 3 respectively; The number of coordinates in the final point candidate set is set as n , n = n 1+ n 2+ n 3; according to whether the text line candidate set is empty, the distance parameter of each coordinate is obtained, and the distance decay term of each coordinate is calculated according to the distance parameter of each coordinate; based on the distance decay term of each coordinate, the normalized weight of each coordinate is calculated; S6: According to the normalized weight of each coordinate, combine a Gaussian kernel function, calculate the Gaussian kernel surface prediction probability of each coordinate in the spatial constraint ring, and take the coordinate represented by the maximum value of the Gaussian kernel surface prediction probability as the final user position; Step S5 comprises the following steps: S5.1: Obtain the distance parameter of each coordinate; The first method: when the text line candidate set When it is an empty set, the first k Distance parameters of each coordinate d k Set value; The second way: when the text line candidate set is not empty, the first k coordinate is the minimum value of the shortest distance between the first coordinate and all line type vectors in the text line candidate set k , and the distance parameter of the first d coordinate is the minimum value of the shortest distance between the first k coordinate and all line type vectors in the text line candidate set S5.2: Calculate the distance decay term of each coordinate; the formula is: ； wherein d b represents a set buffer distance; d k represents k a distance parameter for the s k represents a distance decay term for the k coordinate; S5.3: Calculate the basic weight of each coordinate; according to the basic weight, calculate the weight of each coordinate before normalization; specifically comprising the following steps: S5.3.1: According to the point candidate set from which each coordinate is derived, combine the set weight of the corresponding point candidate set to calculate the basic weight of the coordinate; the formula is: ； wherein, represents the final point candidate set coordinates, derived from the text point candidate set at the base weight; represents the final point candidate set coordinates, derived from the image point candidate set at the base weight; represents the final point candidate set coordinates, derived from the check-in location point candidate set at the base weight; M 1, M 2 and M 3 represent the set weights of the text point candidate set , the image point candidate set and the check-in location point candidate set respectively; n 1, n 2 and n 3 represent the number of coordinates in the final point candidate set , derived from the text point candidate set , the image point candidate set and the check-in location point candidate set respectively; if n 1, n 2 or n 3 is 0, the corresponding base weight is not available; S5.3.2: Obtain the final point candidate set through the operation of S5.3.1 The base weight of the middle coordinate is set to k ;​ S5.4: Calculate the weight before the normalization of the i-th coordinate k ; The formula is: ； wherein represents k the base weight of the th coordinate; k the weight of the s th coordinate before normalization; k represents the distance decay term of the k th coordinate; S5.5: Calculate the normalized weight of the i-th coordinate; k S5.5: Calculate the normalized weight of the i-th coordinate; The formula is: ； wherein represents the weight of the normalized coordinate; k represents the weight of the normalized coordinate; j represents the weight of the normalized coordinate.​ 2. The positioning optimization method based on multi-modal data according to claim 1, characterized in that, Step S4 comprises the following steps: S4.1: Construct a preselected point candidate set, and the formula is: ； wherein represents a union operation; represents a set of text point candidates; represents a set of picture point candidates; represents a set of check-in location point candidates; S4.2: Screen the coordinates in the preselected point candidate set; specifically comprising the following steps: S4.2.1: If the text line candidate set Candidate set of text All are empty sets, so the candidate set of pre-selected points will be... The bounding rectangle formed by all coordinates in the bounding rectangle serves as the spatial constraint circle. S No need to remove the candidate set of pre-selected points The data in the sample retains the candidate set of pre-selected points. Data in the middle; S4.2.2: If the text line candidate set is empty, the text face candidate set is empty; then a spatial constraint circle is established: ; if for the pre-selected point candidate set the i th coordinate x i , y i ) is in the interval ，i [1, m ], and is an integer; m denotes the total number of coordinates in the pre-selected point candidate set ; if holds, the i th coordinate x i , y i ) is kept; if does not hold, the i th coordinate x i , y i ) is deleted; S4.2.3: If the text line candidate set is not empty, the text face candidate set is empty, the spatial constraint circle is established as: ； wherein dist (*, *) denotes the shortest distance of a corresponding coordinate x,y ) to a line vector of a line candidate set d b denotes a set buffer distance;​ For a preselected set of point candidates th coordinate i x i y i ，i ∈[1, m ] and is an integer; if th coordinate i x i y i th coordinate i x i y i is deleted.​​​​​​​​ S4.2.4: If the set of text line candidates and the set of text face candidates are both non-empty sets, then the set of spatial constraints is defined as: ； For the pre-selected point candidate set In the middle i of the coordinates x i , y i ) ，i ∈[1, m ] and is an integer; if or is true, the coordinate is retained; if not, the coordinate is deleted, forming the final point candidate set .

3. The positioning optimization method based on multi-modal data according to claim 1, characterized in that, Step S6 comprises the following steps: S6.1: Calculate spatial constraint circle S Any point in the middle and the final point candidate set Gaussian kernel surface prediction probability of all coordinates; formula: ； wherein denotes any point in the spatial constraint circle S ; denotes any point in the spatial constraint circle S and the final point candidate set all coordinates; the Gaussian kernel surface prediction probability; K (*) denotes the Gaussian kernel function; h denotes the Gaussian kernel bandwidth; x k , y k denotes the coordinate in the final point candidate set k ; Wherein, the Gaussian kernel function is represented as: ； wherein r denotes a Gaussian kernel variable; S6.2: The coordinate represented by the maximum value of the Gaussian kernel surface prediction probability is taken as the final user position; the formula is: ； In the formula, represents the final user position; represents the coordinate with the maximum probability.

Images