Personalized scenic spot recommendation method based on large model picture content feature enhancement

Abstract

The invention discloses a personalized scenic spot recommendation method based on large model picture content feature enhancement, which comprises the following steps: 1, data set acquisition and processing: acquiring a tourism data set with a geographic mark picture, and inversely coding geographic information of the picture to obtain a user-scenic spot interaction matrix; 2, picture content feature extraction: performing data enhancement by using a large model, and converting a picture into interpretable text semantic information; 3, establishing a prediction preference model: adopting a model based on Wideamp; a recommendation model of Deep variants captures a direct relationship between features; and 4, obtaining preference values of the user for all scenic spots from the prediction preference model, sorting the preference values in a descending order, and selecting and recommending the scenic spots corresponding to the top preference values to the user. According to the method, geographic information, time information and picture content attribute information extracted by a large model are integrated, and the design is based on Widamp; according to the recommendation model of the Deep variant, the accuracy and individuation level of scenic spot recommendation are effectively improved.

Description

Technical Field

[0001] The present invention belongs to the field of image processing based on computer vision technology, and mainly involves a personalized scenic spot recommendation method based on reverse geocoding, large model data enhancement, graph neural network and feature intersection. Background Art

[0002] In recent years, with the rapid development of the Internet, many social platforms represented by Flickr have launched image sharing services. Users can upload pictures associated with locations, resulting in the emergence of a large number of pictures with text tags, timestamps and geographic references on the Internet. These pictures not only record the daily lives of users, but also provide researchers with rich data resources. Different from other online media, geotagged pictures connect geographic information, time information and visual information together to become the digital footprint of the photographer. By analyzing the time and geographic information of picture upload, the user's spatiotemporal movement trajectory can be obtained, which can effectively alleviate the common data sparsity problem in traditional geo-tag-based POI (Point Of Interest) recommendations and provide important contextual information for predicting user visits to points of interest.

[0003] Existing POI recommendations based on geo-tagged images usually only use the visual features and geo-tagged information of the image as auxiliary information for the recommendation task, but do not make full use of the deep entity information and atmosphere and emotional information in the image, and do not fully consider the implicit customer preferences for the content characteristics of tourist attractions in social media data, thus reducing the diversity and accuracy of recommendations, and thus affecting user experience. For example, for a single user, a user who posts many rock climbing and diving pictures is more likely to visit adventurous and exciting attractions; for different users, the attractions they have visited may be different, but comparing the pictures may reveal that the content of the attractions is extremely similar, which shows that even in different locations, users may be attracted to similar types of experiences.

[0004] At the same time, most of the existing image visual feature extraction methods are based on deep neural networks (DNNs). Deep neural networks are usually regarded as "black box" models. It is difficult to intuitively understand their internal working mechanisms and decision-making processes. The features are not interpretable and the correlation between different features cannot be captured. This is a significant limitation for POI recommendation application scenarios that require transparency and credibility. At the same time, many geo-tagged images on social platforms currently contain a lot of redundant information, such as low-quality images and repeated content. If the representation of these images is directly extracted, it will put a lot of pressure on the model and it will be difficult to identify specific user preferences. Redundant information may cause the model to overfit, affecting the accuracy and personalization of POI recommendations. Summary of the invention

[0005] In view of the many problems existing in the above-mentioned prior art, the present invention proposes a personalized scenic spot recommendation method based on large model image content feature enhancement, in order to effectively improve the accuracy and personalization level of scenic spot recommendations, thereby improving the recommendation accuracy and explainability.

[0006] The present invention adopts the following technical solutions to solve the technical problems:

[0007] The personalized scenic spot recommendation method based on large model picture content feature enhancement of the present invention is characterized in that it is carried out according to the following steps:

[0008] Step 1: Get the i-th user A collection of geotagged images taken ,in, represents the i-th user The jth picture taken, represents the i-th user The total number of pictures taken, , Represents a collection of users;

[0009] Step 2: The geographic coordinates of the jth picture are inversely encoded to obtain the scenic spot , thus obtaining the i-th user A collection of attractions , , L represents the scenic spot collection;

[0010] Step 3: Use the large model to and the kth attraction Data augmentation is performed on the images to convert the image information into interpretable multi-attribute textual semantic information;

[0011] Step 4: Build a wide-end preference prediction network, perform feature cross-talk on the attribute-level information extracted from the image, capture the relationship between different features, and obtain the i-th user For the kth attraction Preference ;

[0012] Step 5: Build a Deep Preference Prediction Network and as well as and Process and get the i-th user For the kth attraction Preference ;

[0013] Step 6: Use formula (13) to establish the i-th user of the Wide & Deep end joint prediction For the kth attraction Preference :

[0014] (13)

[0015] In formula (13), represents the weight that controls the relative importance between the two model outputs, is the deviation term, Convert the output to Functions in the interval ;

[0016] Step 7: Use formula (14) to construct the logistic regression loss function L, and train the wide preference prediction network and the deep preference prediction network until the loss function converges, thereby obtaining the optimal preference prediction model;

[0017] (14)

[0018] In formula (14), represents the i-th user Have you visited the kth attraction? The true label of

[0019] Step 8: Output the optimal preference prediction model for the i-th user Sort the preference values ​​in the optimal preference set of all attractions in descending order, and recommend the attractions corresponding to the top preference values ​​to the i-th user. .

[0020] The personalized scenic spot recommendation method based on large model image content feature enhancement described in the present invention is also characterized in that step 2 is performed according to the following steps:

[0021] Step 2.1: Calculate using formula (1) Attractions The geographical distance between :

[0022] (1)

[0023] In formula (1), represents the radius of the Earth, and They are pictures and The latitude of the attraction, and are the longitudes of the pictures and attractions respectively;

[0024] Step 2.2, set the distance threshold Δ, if , then Add attractions Otherwise, assign j+1 to j and return to repeat step 2.1 to get the i-th user A collection of attractions , , Represents the set of attractions for all users.

[0025] Further, the step 3 is performed as follows:

[0026] Step 3.1: Use the multimodal visual model GPT4V to Perform data enhancement to obtain the i-th user Entity text collection and atmosphere emotion text collection ,in, represents the i-th user The jth picture taken The entity description text extracted from represents the i-th user The jth picture taken The atmosphere and emotion description text extracted from

[0027] Step 3.2: Use the multimodal visual model GPT4V to identify the kth scenic spot Picture collection Perform data enhancement to obtain the kth scenic spot Entity text collection and atmosphere emotion text collection ,in, represents the kth scenic spot The qth picture of The entity description text extracted from represents the kth scenic spot The qth picture of The atmosphere and emotional description text extracted from represents the kth scenic spot Total number of pictures;

[0028] Step 3.3: If the i-th user Photographed the kth scenic spot , then it means the i-th user Visited the kth attraction , then let the i-th user Interaction vector with attractions The element in row i and column k of , otherwise, let , thus obtaining the interaction matrix , for The number of users in for The number of scenic spots in the , where T represents transpose;

[0029] Step 3.4: Set the i-th user and the kth attraction The ID is converted into a continuous feature vector to obtain the ID feature embedding vector of the i-th user and the ID feature embedding vector of the kth scenic spot ; For the kth scenic spot The longitude and latitude information is normalized and embedded to obtain the geographic feature embedding vector of the kth scenic spot .

[0030] Further, step 4 is performed as follows:

[0031] Step 4.1: Use equation (2) and equation (3) to get the i-th user The entity feature embedding vector and the i-th user The atmosphere emotional feature embedding vector :

[0032] (2)

[0033] (3)

[0034] In formula (2) and formula (3), represents the attention mechanism network;

[0035] Step 4.2: Use equations (4) and (5) to get the k-th scenic spot The entity feature embedding vector and the kth attraction The atmosphere emotional feature embedding vector :

[0036] (4)

[0037] (5)

[0038] Step 4.3: Use formula (6) to establish the prediction preference function of the Wide end :

[0039] (6)

[0040] In formula (6), represents the global bias term, Represents the i-th user The two first-order feature weights of Respectively represent the kth scenic spot The two first-order feature weights of represents the i-th user The second-order cross-feature weights of represents the kth scenic spot The second-order cross-feature weights of Represents the i-th user The two second-order feature weights of Respectively represent the kth scenic spot The two second-order feature weights of .

[0041] Further, the step 5 is performed as follows:

[0042] Step 5.1: Define the current layer of the graph neural network as the nth layer, the total number of layers is N, and initialize n=1. Define and initialize the i-th user Embedding vector at level n-1 , the kth attraction The embedding vector at the n-1th layer is ;

[0043] Step 5.2: Use equations (8) and (9) to get the i-th user In the Embedding vector of the layer and the kth attraction In the Embedding vector of the layer :

[0044] (8)

[0045] (9)

[0046] In formula (8) and formula (9), and Represents the i-th user The number of visited attractions and the kth attraction The number of users accessed;

[0047] Step 5.3: Use equations (10) and (11) to get the i-th user The final embedding vector and the kth attraction The final embedding vector :

[0048] (10)

[0049] (11)

[0050] In formula (10) and formula (11), Indicates The weight coefficient of the layer;

[0051] Step 5.4: Use formula (12) to establish the i-th user predicted by the Deep side For the kth attraction The preference function :

[0052] (12).

[0053] An electronic device of the present invention includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the personalized scenic spot recommendation method, and the processor is configured to execute the program stored in the memory.

[0054] The present invention provides a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and the computer program executes the steps of the personalized scenic spot recommendation method when the computer program is executed by a processor.

[0055] Compared with the prior art, the present invention has the following beneficial effects:

[0056] 1. The present invention coordinates geographic tag information, shooting time information and visual attribute information extracted from images, uses a large model to convert images into interpretable textual semantic information, and uses a predictive preference model based on a Wide&Deep architecture variant to better capture the characteristics of tourist destinations and users' personal interests, thereby providing more accurate and personalized tourist attraction recommendation services.

[0057] 2. The present invention uses inverse coding technology to map the user's geo-tagged image information into the interaction information between the user and the scenic spot, fully mining the user's historical behavior data to establish a more accurate user portrait, effectively alleviating the data sparsity problem existing in the traditional tourist attraction recommendation, and improving the accuracy of the recommendation system and user experience. It not only enhances the connection between users and service providers, but also promotes the digital transformation of related industries and the improvement of service quality.

[0058] 3. The present invention uses a large model to extract deep entity information and atmosphere and emotion information from images, converts image information into more interpretable text information, deeply mines the entity information and atmosphere and emotion information in images, and effectively solves the "black box problem" existing in traditional image visual feature extraction methods, thereby more accurately understanding user needs and providing personalized scenic spot recommendation service suggestions, thereby improving user experience.

[0059] 4. The present invention uses the Wide&Deep architecture variant to establish a predictive preference model. The Wide end uses a feature crossover algorithm to capture the association between different attribute features, and the Deep end uses graph neural network technology to extract the representation of users and attractions. It effectively captures the interactive information between users and attractions while providing explainable POI recommendations, thereby providing a theoretical basis for tourist attraction recommendations. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 It is the overall flow chart of the present invention;

[0061] Figure 2 It is the overall flow chart of the prediction preference model of the present invention. DETAILED DESCRIPTION

[0062] In this embodiment, a personalized scenic spot recommendation method based on large model image content feature enhancement includes: 1. Obtaining a tourism data set with geo-tagged images, inversely encoding the user's image geographic information, and obtaining a user-attraction interaction matrix; 2. Based on the data enhancement of the large model, converting the image into interpretable text semantic information, and obtaining different attribute feature matrices of users and attractions respectively; 3. Establishing a prediction preference model based on Wide & Deep variants to capture the direct relationship between features, and using the logistic regression loss function to update the prediction preference model, so as to obtain the user's preference value for all attractions in the final prediction preference model, and after sorting the preference values ​​in descending order, selecting the attractions corresponding to the top preference values ​​and recommending them to the user. Specifically, if Figure 1 As shown, the method is performed in the following steps:

[0063] Step 1: Get the i-th user A collection of geotagged images taken ,in, represents the i-th user The jth picture taken, represents the i-th user The total number of pictures taken, , Represents a collection of users;

[0064] Step 1.1, initialize the URL list;

[0065] Step 1.2: Call the API to obtain a large amount of product information stored in XML format;

[0066] Step 1.3, parse the XML file, get the seed list, and store the parsed results in the database;

[0067] Step 1.4: After obtaining the seed list of product names, filter and remove duplicates from the list;

[0068] Step 1.5: If you need to expand the URL list, continue to step 1.2, otherwise obtain the user set , picture collection and the geographical coordinates of the image .

[0069] Step 2: The geographic coordinates of the jth picture are inversely encoded to obtain the scenic spot , L represents the scenic spot collection;

[0070] Step 2.1: Calculate using formula (1) Attractions The geographical distance between :

[0071] (1)

[0072] In formula (1), represents the radius of the Earth, and They are pictures and The latitude of the attraction, and are the longitudes of the pictures and attractions respectively;

[0073] Step 2.2, set the distance threshold Δ, if , then Add attractions Otherwise, assign j+1 to j and return to repeat step 2.1 to get the i-th user A collection of attractions , , Represents the set of attractions for all users.

[0074] Step 3: Use the large model to perform data enhancement and convert the image information into interpretable multi-attribute text semantic information;

[0075] Step 3.1: Use the multimodal visual model GPT4V to Perform data enhancement to obtain the i-th user Entity text collection and atmosphere emotion text collection ,in, represents the i-th user The jth picture taken The entity description text extracted from represents the i-th user The jth picture taken The atmosphere and emotion description text extracted from

[0076] Step 3.2: Use the multimodal visual model GPT4V to identify the kth scenic spot Picture collection Perform data enhancement to obtain the kth scenic spot Entity text collection and atmosphere emotion text collection ,in, represents the kth scenic spot The qth picture of The entity description text extracted from represents the kth scenic spot The qth picture of The atmosphere and emotional description text extracted from represents the kth scenic spot The total number of pictures.

[0077] Step 3.3: If the i-th user Photographed the kth scenic spot , then it means the i-th user Visited the kth attraction , then let the i-th user Interaction vector with attractions The element in row i and column k of , otherwise, let , thus obtaining the interaction matrix , for The number of users in for The number of scenic spots in the , where T represents transpose;

[0078] Step 3.4: Set the i-th user and the kth attraction The ID is converted into a continuous feature vector to obtain the ID feature embedding vector of the i-th user and the ID feature embedding vector of the kth scenic spot ; For the kth scenic spot The longitude and latitude information is normalized and embedded to obtain the geographic feature embedding vector of the kth scenic spot. .

[0079] Step 4: Build a wide-end preference prediction network, perform feature cross-talk on the attribute-level information extracted from the image, capture the relationship between different features, and obtain the i-th user For the kth attraction Preference ;

[0080] Step 4.1: Use equation (2) and equation (3) to get the i-th user The entity feature embedding vector and the i-th user The atmosphere emotional feature embedding vector :

[0081] (2)

[0082] (3)

[0083] In formula (2) and formula (3), Represents the attention mechanism network.

[0084] Step 4.2: Use equations (4) and (5) to get the k-th scenic spot The entity feature embedding vector and the kth attraction The atmosphere emotional feature embedding vector :

[0085] (4)

[0086] (5)

[0087] Step 4.3: Use formula (6) to establish the prediction preference function of the Wide end :

[0088] (6)

[0089] In formula (6), represents the global bias term, Respectively represent users and attractions The first-order feature weights of Indicates user The second-order cross-feature weights of Indicate attractions The second-order cross-feature weights of Respectively represent users and attractions The second-order feature weights of .

[0090] Step 5: Build a Deep Preference Prediction Network and as well as and Process and get the i-th user For the kth attraction Preference ;

[0091] Step 5.1: Define the current layer of the graph neural network as the nth layer, the total number of layers is N, and initialize n=1. Let the i-th user The initial embedding vector , the kth attraction The initial embedding vector of ;

[0092] Step 5.2: Use equations (8) and (9) to get the i-th user In the Embedding vector of the layer and the kth attraction In the Embedding vector of the layer :

[0093] (8)

[0094] (9)

[0095] In formula (8) and formula (9), and Represents the i-th user The number of visited attractions and the kth attraction The number of users accessing Indicates user The embedding vector of the n-1th layer, Indicate attractions Embedding vector at layer n-1.

[0096] Step 5.3: Use equations (10) and (11) to get the i-th user The final embedding vector and the kth attraction The final embedding vector :

[0097] (10)

[0098] (11)

[0099] In formula (10) and formula (11), Indicates user No. Embedding vector of the layer and attractions No. Embedding vector of the layer The weight coefficient of

[0100] Step 5.4: Use formula (12) to establish the i-th user predicted by the Deep side For the kth attraction The preference function :

[0101] (12).

[0102] Step 6: Use formula (13) to establish the i-th user of the Wide & Deep end joint prediction For the kth attraction Preference :

[0103] (13)

[0104] In formula (13), represents the weight that controls the relative importance between the two model outputs, is the deviation term, Convert the output to Function in the interval .

[0105] Step 7: Use formula (14) to construct the logistic regression loss function L, and train the wide preference prediction network and the deep preference prediction network until the loss function converges, thereby obtaining the optimal preference prediction model;

[0106] (14)

[0107] In formula (14), represents the i-th user Have you visited the kth attraction? The real label.

[0108] Step 8: Output the optimal preference prediction model for the i-th user Sort the preference values ​​in the optimal preference set of all attractions in descending order, and recommend the attractions corresponding to the top preference values ​​to the i-th user. .

[0109] In this embodiment, an electronic device includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the above method, and the processor is configured to execute the program stored in the memory.

[0110] In this embodiment, a computer-readable storage medium stores a computer program on the computer-readable storage medium, and the computer program executes the steps of the above method when executed by a processor.

Claims

1. A personalized scenic spot recommendation method based on large model image content feature enhancement, characterized in that: The steps are as follows: Step 1: Get the i-th user A collection of geotagged images taken ,in, represents the i-th user The jth picture taken, represents the i-th user The total number of pictures taken, , Represents a collection of users; Step 2: The geographic coordinates of the jth picture are inversely encoded to obtain the scenic spot , thus obtaining the i-th user A collection of attractions , , L represents the scenic spot collection; Step 3: Use the large model to and the kth attraction Data augmentation is performed on the images to convert the image information into interpretable multi-attribute textual semantic information; Step 4: Build a wide-end preference prediction network, perform feature cross-talk on the attribute-level information extracted from the image, capture the relationship between different features, and obtain the i-th user For the kth attraction Preference ; Step 5: Build a Deep Preference Prediction Network and as well as and Process and get the i-th user For the kth attraction Preference ; Step 6: Use formula (13) to establish the i-th user of the Wide & Deep end joint prediction For the kth attraction Preference : (13) In formula (13), represents the weight that controls the relative importance between the two model outputs, is the deviation term, Convert the output to Functions in the interval ; Step 7: Use formula (14) to construct the logistic regression loss function L, and train the wide preference prediction network and the deep preference prediction network until the loss function converges, thereby obtaining the optimal preference prediction model; (14) In formula (14), represents the i-th user Have you visited the kth attraction? The true label of Step 8: Output the optimal preference prediction model for the i-th user Sort the preference values ​​in the optimal preference set of all attractions in descending order, and recommend the attractions corresponding to the top preference values ​​to the i-th user. .

2. The personalized scenic spot recommendation method based on large model image content feature enhancement according to claim 1 is characterized in that: Described step 2 is carried out as follows: Step 2.1: Calculate using formula (1) Attractions The geographical distance between : (1) In formula (1), represents the radius of the Earth, and They are pictures and The latitude of the attraction, and are the longitudes of the pictures and attractions respectively; Step 2.2, set the distance threshold Δ, if , then Add attractions Otherwise, assign j+1 to j and return to repeat step 2.1 to get the i-th user A collection of attractions , , Represents the set of attractions for all users.

3. The personalized scenic spot recommendation method based on large model image content feature enhancement according to claim 2 is characterized in that: The step 3 is carried out as follows: Step 3.1: Use the multimodal visual model GPT4V to Perform data enhancement to obtain the i-th user Entity text collection and atmosphere emotion text collection ,in, represents the i-th user The jth picture taken The entity description text extracted from represents the i-th user The jth picture taken The atmosphere and emotion description text extracted from Step 3.2: Use the multimodal visual model GPT4V to identify the kth scenic spot Picture collection Perform data enhancement to obtain the kth scenic spot Entity text collection and atmosphere emotion text collection ,in, represents the kth scenic spot The qth picture of The entity description text extracted from represents the kth scenic spot The qth picture of The atmosphere and emotional description text extracted from represents the kth scenic spot Total number of pictures; Step 3.3: If the i-th user Photographed the kth scenic spot , then it means the i-th user Visited the kth attraction , then let the i-th user Interaction vector with attractions The element in row i and column k of , otherwise, let , thus obtaining the interaction matrix , for The number of users in for The number of scenic spots in the , where T represents transpose; Step 3.4: Set the i-th user and the kth attraction The ID is converted into a continuous feature vector to obtain the ID feature embedding vector of the i-th user and the ID feature embedding vector of the kth scenic spot ; For the kth scenic spot The longitude and latitude information is normalized and embedded to obtain the geographic feature embedding vector of the kth scenic spot .

4. The personalized scenic spot recommendation method based on large model image content feature enhancement according to claim 3 is characterized in that: The step 4 is carried out as follows: Step 4.1: Use equation (2) and equation (3) to get the i-th user The entity feature embedding vector and the i-th user The atmosphere emotional feature embedding vector : (2) (3) In formula (2) and formula (3), represents the attention mechanism network; Step 4.2: Use equations (4) and (5) to get the k-th scenic spot The entity feature embedding vector and the kth attraction The atmosphere emotional feature embedding vector : (4) (5) Step 4.3: Use formula (6) to establish the prediction preference function of the Wide end : (6) In formula (6), represents the global bias term, Represents the i-th user The two first-order feature weights of Respectively represent the kth scenic spot The two first-order feature weights of represents the i-th user The second-order cross-feature weights of represents the kth scenic spot The second-order cross-feature weights of Respectively represent the i-th user The two second-order feature weights of Respectively represent the kth scenic spot The two second-order feature weights of .

5. The personalized scenic spot recommendation method based on large model image content feature enhancement according to claim 4 is characterized in that: The step 5 is carried out as follows: Step 5.1: Define the current layer of the graph neural network as the nth layer, the total number of layers is N, and initialize n=1. Define and initialize the i-th user Embedding vector at level n-1 , the kth attraction The embedding vector at the n-1th layer is ; Step 5.2: Use equations (8) and (9) to get the i-th user In the Embedding vector of the layer and the kth attraction In the Embedding vector of the layer : (8) (9) In formula (8) and formula (9), and Respectively represent the i-th user The number of visited attractions and the kth attraction The number of users accessed; Step 5.3: Use equations (10) and (11) to get the i-th user The final embedding vector and the kth attraction The final embedding vector : (10) (11) In formula (10) and formula (11), Indicates The weight coefficient of the layer; Step 5.4: Use formula (12) to establish the i-th user predicted by the Deep side For the kth attraction The preference function : (12)。 6. An electronic device, comprising a memory and a processor, characterized in that: The memory is used to store a program that supports the processor to execute the personalized scenic spot recommendation method according to any one of claims 1 to 5, and the processor is configured to execute the program stored in the memory.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the personalized scenic spot recommendation method according to any one of claims 1 to 5 are executed.

Images