Multi-modal interactive tourism recommendation method, system, equipment and medium

Abstract

The invention provides a multi-modal interactive tourism recommendation method, system and device and a medium, and belongs to the technical field of tourism interaction, and the method comprises the steps: constructing a user tourism portrait based on the combination of clustering and anomaly detection; according to the clustering result and the user behavior data, constructing user portrait labels and feature descriptions, and configuring the constructed user portraits into a multi-modal interaction module; an interaction interface unit, a voice recognition unit, an image recognition unit and a gesture control and interaction unit are configured based on the multi-mode interaction module; the multi-mode interaction module generates a VR panorama based on voice information, image information and gestures input by a user, and feeds back the VR panorama to the interaction interface unit in cooperation with voice and touch. An interactive interface is provided through a virtual reality technology, so that a user can perform customization operation of using the embodiment without perception, the user experience is improved, and more intuitive and immersive destination and recommendation place information experience is obtained.

Description

Technical Field

[0001] The present invention belongs to the field of tourism interaction technology, and in particular relates to a multi-modal interactive tourism recommendation method, system, device and medium. Background Art

[0002] Traveling has become a popular leisure activity. In order to meet the needs of tourists, various tourism service providers have begun to seek more accurate and personalized service methods. In this context, data-driven personalized tourism recommendation systems have emerged and become one of the key technologies to improve tourism experience and service quality.

[0003] At present, most of the travel recommendation systems are based on the user's historical behavior data, such as browsing history, purchase history, etc., and provide users with personalized travel suggestions through collaborative filtering, content recommendation and other algorithms. However, these methods often only consider the user's single behavior pattern, ignoring the multimodal characteristics of the user during the travel process, and cannot meet the user's demand for travel recommendations. In addition, the existing travel recommendation system lacks interactivity, and searches for travel destinations and related travel information through keyboard input commands, lacking interactivity with users. Summary of the invention

[0004] The present invention provides a multimodal interactive tourism recommendation method, which can dynamically display tourism information, realize dynamic interaction with the user's current state and environment, and enhance the richness of user experience.

[0005] Methods include: Construct user travel profiles based on clustering and anomaly detection; Construct user portrait labels and feature descriptions based on clustering results and user behavior data, and configure the constructed user portraits into the multimodal interaction module; Based on the multimodal interaction module, an interactive interface unit, a speech recognition unit, an image recognition unit, and a gesture control and interaction unit are configured; The multimodal interaction module generates a VR panorama based on the voice information, image information and gestures input by the user, and provides voice and tactile feedback to the interaction interface unit.

[0006] It should be further explained that the steps of constructing a user travel profile based on clustering and anomaly detection also include: Obtain the user's historical travel data, which includes the user's travel destination, stay time, consumption information, and travel trajectory information; Preprocessing of historical tourism data; Group historical travel data based on clustering algorithms to identify travel behavior patterns corresponding to different user groups; Integrate the cluster analysis results to construct user travel portraits.

[0007] It is further necessary to explain that, in the method, after obtaining the clustering results, an anomaly detection algorithm is used to identify the historical tourism data for detection to obtain abnormal behaviors.

[0008] It should be further explained that the step of configuring the constructed user portrait into the multimodal interaction module also includes: Extract the user's historical travel data and convert it into a numerical vector; Configure the extraction of user travel key features, define user travel portrait tags based on the extracted user travel key features, and describe the features of each user travel portrait tag; Configure the user's key travel features into the travel application scenario of the multimodal interaction module, and configure the interaction method based on the user's travel portrait tags; Among them, the configuration item PZ of the user's key tourism features in the tourism application scenario of the multimodal interaction module is: PZ = σ(QxL yi + QyLyi−1 + pz); Among them, yi is the extracted historical travel data of the i-th user, QxL, QyL are weight matrices, pz is the bias term, and σ is the activation function.

[0009] It should be further explained that the method also includes, based on the user's travel profile, combining the following travel recommendation algorithm, recommending travel data to the user; User recommended travel data Sv = f (L1, L2, L3, Ln) f() represents the feature vector fusion function, Ln represents the user's key travel features, and n represents the number of user's key travel features.

[0010] f() is used to fuse two or more user travel key features into one user travel key feature. The fusion method includes but is not limited to sequentially combining into a longer vector, or weighted fusion, or other fusion methods.

[0011] It should be further explained that the gesture control methods include: Use VR glove sensors to collect user hand motion data; Extracting gesture features from user hand motion data; Classify and recognize gesture features using preset algorithms; The recognized gesture actions are mapped to a predefined instruction set to realize the association between gestures and operations, and an interactive tourism interface corresponding to the gesture actions is displayed in the interactive interface unit.

[0012] It should be further explained that the preset algorithm in the step of using the preset algorithm to classify and identify the gesture features is: Interactive tourism interface Mc=∑p∑qT(r+p,j+q)H(p,q); The interactive tourism interface Mc is calculated by sliding the convolution kernel H on the input image T; r and j represent the row index and column index on the user's travel key features; r and j traverse all user travel key features; p and q represent the row index and column index of the convolution kernel H respectively; In the process of gesture feature recognition, p and q will traverse all elements of the convolution kernel. H(p,q) represents the element value of the convolution kernel H in the pth row and qth column; the convolution kernel is a weight matrix used to extract the features of the user's travel portrait; T(r+p,j+q) represents the element value of the user's travel key feature at the position corresponding to the convolution kernel H; When the convolution kernel slides on the user's key travel features, T(r+p,j+q) points to different positions on the user's key travel features accordingly. By multiplying the convolution kernel with the corresponding elements of the input image and summing them, the interactive travel interface Mc is obtained.

[0013] The present application also provides a multimodal interactive tourism recommendation system, the system comprising: The portrait construction module is used to construct user travel portraits based on the combination of clustering and anomaly detection; The portrait configuration module is used to construct user portrait labels and feature descriptions based on clustering results and user behavior data, and configure the constructed user portraits into the multimodal interaction module; The multimodal interaction module is configured with an interaction interface unit, a speech recognition unit, an image recognition unit, and a gesture control and interaction unit; The multimodal interaction module generates a VR panorama based on the voice information, image information and gestures input by the user, and provides voice and tactile feedback to the interaction interface unit.

[0014] According to another embodiment of the present application, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the multimodal interactive tourism recommendation method when executing the program.

[0015] According to another embodiment of the present application, a storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the multimodal interactive tourism recommendation method are implemented.

[0016] It can be seen from the above technical solutions that the present invention has the following advantages: The multimodal interactive travel recommendation method provided by the present invention can construct a more detailed user portrait by collecting and analyzing the user's historical travel data, including travel destinations, stay time, consumption habits, etc. It can more accurately grasp the user's preferences and needs, thereby providing more accurate recommendation services.

[0017] The present invention also uses a clustering algorithm to group users and identify different user groups and their travel behavior patterns, which enables the recommendation system to provide more personalized recommendation solutions for different types of users and meet the differentiated needs of users.

[0018] The present invention can meet the diverse interaction needs of users through a multi-modal input interactive interface. Users can choose different input methods such as voice, image, gesture, etc. according to their own habits and preferences, easily query travel information, and obtain personalized recommendation results. The fun and immersion of interaction are increased, thereby improving the user experience.

[0019] The present invention closely combines multimodal interaction with a personalized recommendation system, and generates more accurate and personalized recommendation results by deeply integrating user preference factors. The system can integrate user input information such as voice, images, gestures, and multi-dimensional information such as historical behavior data, interest preferences, and travel habits to fully understand the user's query intentions and needs, thereby providing users with travel recommendations that are more in line with their actual needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solution of the present invention, the accompanying drawings required for use in the description will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0021] Figure 1 A flow chart of the method for multimodal interactive tourism recommendation; Figure 2 Schematic diagram of a multimodal interactive tourism recommendation system; Figure 3 Schematic diagram of an electronic device. DETAILED DESCRIPTION

[0022] The multimodal interactive tourism recommendation method proposed in the present invention mainly solves the problems of insufficient satisfaction of user personalized needs, low recommendation accuracy and delayed abnormal behavior recognition in traditional tourism recommendation systems.

[0023] The present invention supports an interactive interface with multi-modal input, and can guide users to use various interactive methods, such as voice, image, gesture, etc. It allows users to query travel information through voice, image, and gesture, and uses voice and image technology to feed back the query results to users, thereby improving interactivity and convenience.

[0024] The present invention can also query related information or make similar recommendations through pictures. The system can identify key elements in the pictures and provide relevant travel suggestions or information accordingly.

[0025] The steps of the multimodal interactive travel recommendation method are described in detail below. For the purpose of illustration rather than limitation, specific details such as specific system structures and technologies are provided to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application can also be implemented in other embodiments without these specific details.

[0026] The phrases such as "one embodiment" or "some embodiments" described in the present application mean that the specific features, structures or characteristics described in the embodiment are included in one or more embodiments of the present application. Therefore, the phrases such as "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments" etc. that appear in different places in the present application do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways.

[0027] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] See also Figure 1 FIG. 1 is a flowchart of a multimodal interactive tourism recommendation method in a specific embodiment, the method comprising: S101: Construct user travel profile based on the combination of clustering and anomaly detection.

[0029] In some embodiments, historical travel data of users is collected from multiple channels. Historical travel data includes, but is not limited to, records of user travel destinations, length of stay at each destination, consumption habits (such as expenditures on dining, accommodation, shopping, etc.), and travel activities participated in (such as sightseeing, experiencing local culture, outdoor adventures, etc.). In order to ensure the comprehensiveness and accuracy of the data, data can be shared and exchanged with major travel platforms, hotels, airlines, etc.

[0030] Conduct preliminary screening and sorting of the collected data. This mainly involves removing duplicate data, filling missing values, and preliminarily identifying and processing abnormal data. For example, further verification and correction will be conducted for stay time or consumption amounts that are clearly beyond a reasonable range.

[0031] Specifically, data cleaning can use a variety of methods to remove noise and irrelevant data. This includes using regular expressions to match and remove irrelevant text information, using domain knowledge bases to correct erroneous data, etc. At the same time, data visualization technology is also used to conduct preliminary exploration of data distribution to discover potential outliers and outliers.

[0032] In the feature extraction process, this embodiment extracts features that are helpful for describing user behavior patterns from the original data according to the needs of the travel recommendation scenario. These features may include the user's travel preferences (such as whether they like natural scenery or urban scenery), consumption level (such as the average cost per trip), and activity participation (such as whether they often participate in outdoor activities, etc.). In order to extract these features, this embodiment uses natural language processing (NLP) technology to perform sentiment analysis and topic extraction on user comments and feedback, and uses statistical analysis methods to perform feature selection and dimensionality reduction on numerical data.

[0033] This embodiment normalizes the extracted features to eliminate the influence of dimensions and value ranges between different features, which helps to improve the performance and accuracy of subsequent clustering algorithms and anomaly detection algorithms.

[0034] As an implementation of the present application, feature extraction is to extract useful information for subsequent cluster analysis and anomaly detection from raw data. The extracted features can reflect the user's travel behavior patterns and consumption habits.

[0035] In the feature extraction process, this embodiment can extract features based on the user's travel destination, stay time, consumption amount, consumption type (such as transportation, accommodation, catering, entertainment, etc.), tour activities (such as sightseeing, shopping, leisure, etc.), etc. For example, the average stay time, total consumption amount, proportion of various types of consumption, etc. of the user at each destination can be calculated to form a series of features that can characterize the user's travel behavior. It can also be based on the characteristics of time series, such as the frequency of user travel, the seasonality of travel time, etc., to more comprehensively reflect the user's travel behavior pattern.

[0036] Since different features may have different dimensions and value ranges, directly using them for cluster analysis may lead to biased results. Therefore, after feature extraction, the data needs to be standardized. There are many methods for data standardization, such as Z-score standardization, Min-Max standardization, etc. Through standardization, the dimensional differences between features can be eliminated, so that different features have the same weight in cluster analysis. In this way, the clustering algorithm can more accurately identify user groups and their behavior patterns.

[0037] In some specific embodiments, after completing the cluster analysis, an anomaly detection algorithm is used to identify user behaviors that are significantly different from regular travel behavior patterns.

[0038] Anomaly detection can use the local anomaly factor (LOF) anomaly detection algorithm to identify outliers in user behavior data. In the cultural and tourism data scenario, this algorithm is used to identify abnormal patterns in user behavior, including abnormally high-frequency search behavior and extremely preferred scenic spot selection. This step of anomaly detection is used in the cultural and tourism scenario to screen out tourist search history and ticket ordering information.

[0039] In this embodiment, the algorithm parameters are adjusted according to the characteristics of tourism industry data to ensure the accuracy and efficiency of anomaly detection. According to the distribution characteristics of cultural and tourism data, an appropriate neighborhood size is selected. For tourist search history, this embodiment marks the same search records of the same user on the same day as abnormal data; for ticket ordering information, this embodiment marks the ticket ordering of the same attraction by the same user within a week as abnormal data (by setting a simple data neighborhood in this way). After setting the neighborhood in this way, it is set by analyzing the distribution density and abnormal points of the data, and adjusted through experiments to optimize the anomaly detection effect. The detected outliers are marked or removed to reduce the impact on subsequent clustering analysis.

[0040] As an anomaly detection method of this embodiment, the IsolationForest algorithm can also be used for anomaly detection. The IsolationForest algorithm constructs multiple isolation trees by randomly selecting features and segmentation values ​​until each leaf node contains only one sample or reaches the maximum depth of the tree. Abnormal points are often isolated at a shallow level and thus identified as abnormalities.

[0041] Specifically, the preprocessed historical tourism data is used to train an anomaly detection model based on the IsolationForest algorithm. During the training process, the model parameters are tuned through cross-validation and grid search techniques to improve the accuracy of anomaly detection.

[0042] After training, the anomaly detection model is applied to real-time tourism data to identify abnormal behaviors that are significantly different from normal patterns. Such abnormal behaviors may include abnormally long stays, abnormally high spending amounts, or abnormally frequent excursions.

[0043] After completing cluster analysis and anomaly detection, the results of the two are integrated to construct a travel-related profile of the user.

[0044] Here, the user group classification obtained by cluster analysis is integrated with the abnormal behavior identified by anomaly detection. For each user, we know the user group to which he belongs and can identify his abnormal behavior. Based on the integrated results, a travel-related portrait is constructed for each user. The portrait not only reflects the user's regular travel behavior pattern (such as preferred travel destinations, consumption habits, etc.), but also highlights his abnormal behavior (such as sudden increase in consumption, unconventional travel routes, etc.). Moreover, with the continuous recording of user travel behavior and the continuous addition of new data, the user's travel portrait is regularly updated. At the same time, by collecting user feedback and actual effect evaluation, the construction method and accuracy of the portrait are continuously optimized.

[0045] S102: Construct user portrait labels and feature descriptions based on clustering results and user behavior data, and configure the constructed user portraits into the multimodal interaction module.

[0046] In some embodiments, labels and feature descriptions are constructed for each user or user group based on clustering results and user behavior data. Among them, the labels can accurately reflect the user's travel preferences and behavioral characteristics. The feature description includes the user's basic information, travel preferences, behavioral characteristics, etc. This embodiment converts the constructed user portrait data into a data format that can be understood and processed by the multimodal interaction module. The converted user portrait data is then configured in the multimodal interaction module. In this way, it can be ensured that the multimodal interaction module can correctly read and understand the user portrait data, and perform corresponding interaction processing based on the data.

[0047] S103: configuring an interactive interface unit, a voice recognition unit, an image recognition unit, and a gesture control and interaction unit based on the multimodal interaction module.

[0048] In some embodiments, an interactive interface unit supporting multi-modal input is configured. The interactive interface unit can guide users to easily use various interactive methods. For example, a clear microphone icon and voice prompt are designed for voice input, and an intuitive photo or upload button is provided for image input.

[0049] Optionally, the interactive interface unit can be compatible and responsive on mobile phones, tablets, and desktop computers. At the same time, the interface should integrate a personalized recommendation module to intelligently recommend travel products or services based on user preferences and historical behaviors.

[0050] In this embodiment, the voice recognition unit of the multimodal interaction module allows the user to query travel information by voice. At the same time, the text-to-speech technology is used to feed back the query results to the user in the form of voice, thereby improving the naturalness and convenience of the interaction.

[0051] The multimodal interaction module also has an image recognition unit, which is computer vision technology (convolutional neural network (CNN)), to achieve automatic recognition and classification of images. Users can query relevant information or make similar recommendations by uploading pictures (such as scenic spot photos, tourism promotion posters, etc.). The system can identify key elements in the picture and provide relevant travel advice or information accordingly.

[0052] For the gesture control and interaction unit, a gesture recognition device is configured. Users can control the display, switching or zooming of tourism information through simple gestures, which increases the fun and immersion of the interaction.

[0053] S104: The multimodal interaction module generates a VR panoramic image based on the voice information, image information and gestures input by the user, and provides voice and tactile feedback to the interaction interface unit.

[0054] In some embodiments, voice information, image information and gesture information input by the user may be received first, and the input information may be pre-processed and feature extracted for subsequent processing and analysis.

[0055] Generate a VR panorama based on the input information and user portrait data. The VR panorama here can truly reflect the travel scenes and experiences expected by the user. Generate corresponding voice and tactile feedback based on the content of the VR panorama and the user's travel preferences and behavioral characteristics. Among them, voice feedback and tactile feedback can guide the user to experience the VR panorama. Output the generated voice and tactile feedback to the interactive interface unit. Ensure that the user can clearly perceive the feedback information through the interactive interface unit and perform corresponding interactive operations based on the feedback information.

[0056] Based on the above-mentioned multimodal interactive tourism recommendation method, a cross-modal fusion system is constructed, which can integrate multiple modules such as speech recognition, image recognition, gesture control, etc., and can realize switching and collaborative work between various unit modules.

[0057] Based on the cross-modal fusion system, when a user searches for a certain scenic spot through voice, the system can automatically switch to the image display mode to display the pictures and detailed introduction of the scenic spot. At the same time, users can also switch between different display modes through gestures, such as switching from picture display to video playback or 3D model display.

[0058] In order to achieve collaborative work between different modes, this embodiment allows the system to integrate this information to generate more comprehensive and accurate recommendation results when users input query information through multiple modes. For example, users can query tourist attractions in a city through voice input, and select their preferred tourism type (such as natural scenery, historical culture, etc.) through gestures. The system can combine this information to recommend tourist attractions that meet the user's needs. The mode switching method can also be adjusted based on user feedback to improve user satisfaction.

[0059] In one embodiment of the present invention, the constructed user portrait is configured to the multimodal interaction module based on the steps. A possible embodiment is given below to illustrate its specific implementation scheme in a non-limiting manner.

[0060] In this embodiment, key user travel features, such as common behavior patterns, interests, and travel destinations of user groups, can be extracted from clustering results and user behavior data.

[0061] User portrait tags are defined based on the extracted features, and each tag is described in detail, including the size of the user group, the length of stay at each destination, consumption habits, and tour activities participated in.

[0062] Configure the multimodal interaction module and select appropriate interaction methods according to the application scenario, such as voice interaction, gesture interaction, and touch interaction. Define the data interface of the multimodal interaction module and import user portrait data into the multimodal interaction module. Integrate user portrait labels and feature descriptions into the algorithm of the multimodal interaction module to provide personalized services and recommendations based on user portraits during the interaction process.

[0063] This embodiment can also perform functional testing on the multimodal interaction module to ensure that the user portrait can be correctly applied. Users can be invited to conduct experience testing, collect feedback, and optimize and improve based on the feedback. In this way, user portrait labels and feature descriptions based on clustering results and user behavior data are constructed and configured in the multimodal interaction module, while using historical monitoring data for model training to accurately identify the experience of multimodal interactive tourism.

[0064] As a preferred method of this embodiment, the historical travel data of the user is extracted and converted into a numerical vector; Configure the extraction of user travel key features, define user travel portrait tags based on the extracted user travel key features, and describe the features of each user travel portrait tag; Configure the user's key travel features into the travel application scenario of the multimodal interaction module, and configure the interaction method based on the user's travel portrait tags; Among them, the configuration item PZ of the user's key tourism features in the tourism application scenario of the multimodal interaction module is: PZ = σ(QxL yi + QyLyi−1 + pz); Among them, yi is the extracted historical travel data of the i-th user, QxL, QyL are weight matrices, pz is the bias term, and σ is the activation function.

[0065] It can be seen that this embodiment converts the user's historical travel data into a numerical vector so that the data can be effectively processed and calculated by the computer. By configuring the extraction of key travel features of users and defining user travel portrait labels, a deep understanding and accurate characterization of user travel preferences can be achieved. The weight matrix and bias term in the formula PZ=σ(QxL yi+QyLyi−1+pz) can be dynamically adjusted according to actual data, so that the system can better adapt to the travel preferences and behavior patterns of different users, and improve the flexibility and robustness of the system.

[0066] On the basis of the above embodiments, in order to further improve the interactivity of the multimodal interactive tourism recommendation method provided in the above embodiments, an implementable method can be used to recommend travel methods and destinations in combination with user preferences. The interactive tourism recommendation method of this embodiment can combine multimodal interaction with personalized recommendations and deeply integrate user preference factors.

[0067] Specifically, the system will not only understand the user's query intention based on input information such as voice, images, gestures, etc., but will also integrate multi-dimensional information such as the user's historical behavior data, interest preferences, travel habits, etc. to generate more accurate and personalized recommendation results.

[0068] As an implementation method of this embodiment, based on the user's travel profile, combined with the following travel recommendation algorithm, travel data is recommended to the user; User recommended travel data Sv = f (L1, L2, L3, Ln) f() represents the feature vector fusion function, Ln represents the user travel key feature, and n represents the number of user travel key features. f() is used to fuse two or more user travel key features into one user travel key feature. The fusion method includes but is not limited to sequentially combining into a longer vector, or weighted fusion, or other fusion methods.

[0069] By integrating the key travel features of users, the travel recommendation algorithm of this embodiment can provide personalized travel data recommendations for each user. Moreover, based on the feature vector fusion function f() of this embodiment, multiple key travel features of users can be effectively combined to form a more comprehensive user travel portrait, thereby improving the accuracy of the recommendation.

[0070] In this embodiment, the execution method of user recommended travel data Sv can extract key features from the user's historical travel data. Use the feature vector fusion function f() to fuse multiple user travel key features into one user travel key feature. This can be achieved by combining them in sequence into longer vectors, weighted fusion or other fusion methods. Based on the fused user travel key features, the travel recommendation algorithm can filter out the travel data that best meets the user's needs from the massive travel data and recommend it to the user.

[0071] In order to improve the interactivity of this embodiment, the gesture control method includes: using a VR glove sensor to collect user hand motion data; extracting gesture features from the user's hand motion data; using a preset algorithm to classify and identify gesture features; mapping the identified gesture actions to a predefined instruction set to achieve the association between gestures and operations, and displaying an interactive tourism interface corresponding to the gesture actions on the interactive interface unit.

[0072] The preset algorithm in the step of using the preset algorithm to classify and identify gesture features is: Interactive tourism interface Mc=∑p∑qT(r+p,j+q)H(p,q).

[0073] The interactive tourism interface Mc is calculated by sliding the convolution kernel H on the input image T; r and j represent the row index and column index on the user's tourism key features; r and j traverse all the user's tourism key features; p and q represent the row index and column index of the convolution kernel H respectively; during the gesture feature recognition process, p and q will traverse all the elements of the convolution kernel. H(p,q) represents the element value of the convolution kernel H on the pth row and qth column; the convolution kernel is a weight matrix used to extract the features of the user's tourism portrait; T(r+p,j+q) represents the element value of the user's tourism key features at the corresponding position of the convolution kernel H; when the convolution kernel slides on the user's tourism key features, T(r+p,j+q) points to different positions on the user's tourism key features accordingly. The interactive tourism interface Mc is obtained by multiplying the convolution kernel with the corresponding elements of the input image and summing them.

[0074] For the interactive tourism interface Mc, an input image T containing user gesture features is obtained. A convolution kernel H is defined to extract the features of the user's tourism portrait. The size and shape of the convolution kernel can be adjusted according to actual needs. The convolution kernel H is slid on the input image T and a convolution operation is performed. For each position (r, j), the sum of the products of the convolution kernel and the corresponding elements of the input image is calculated to obtain the convolution result of the position. The convolution results of all positions are summarized to obtain the interactive tourism interface Mc. This interface reflects the overall situation of the user gesture features in the input image. According to the results of the interactive tourism interface Mc, the gestures are classified and recognized using a preset algorithm or model. According to the results of gesture recognition, the interface responds accordingly, such as adjusting the display content, performing specific operations, etc.

[0075] In this way, by accurately classifying and identifying gesture features, the interactive tourism interface can respond to user instructions more smoothly and operate the interface without complicated key or touch operations. In addition, combined with the user's travel portrait, gesture recognition can provide users with more personalized services.

[0076] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0077] The following is an embodiment of the multimodal interactive tourism recommendation system provided by the embodiments of the present disclosure. The system and the multimodal interactive tourism recommendation methods of the above-mentioned embodiments belong to the same inventive concept. For details not described in detail in the embodiments of the multimodal interactive tourism recommendation system, please refer to the embodiments of the above-mentioned multimodal interactive tourism recommendation method.

[0078] like Figure 2 As shown, the system includes: The portrait construction module is used to construct user travel portraits based on the combination of clustering and anomaly detection.

[0079] The portrait configuration module is used to construct user portrait labels and feature descriptions based on clustering results and user behavior data, and configure the constructed user portraits into the multimodal interaction module.

[0080] The multimodal interaction module is configured with an interaction interface unit, a speech recognition unit, an image recognition unit, and a gesture control and interaction unit.

[0081] The multimodal interaction module generates a VR panorama based on the voice information, image information and gestures input by the user, and provides voice and tactile feedback to the interaction interface unit.

[0082] As for the multimodal interactive tourism recommendation system provided in this embodiment, it can be implemented based on the following preferred methods. Among them, when constructing user tourism-related portraits, use Python programming language for data processing, including data cleaning, encoding, normalization and other operations. Use Pandas library for data processing and analysis. According to the anomaly detection algorithm, write corresponding code implementation. Perform anomaly detection on the data and save the results in a new data set.

[0083] The system of this embodiment can use the clustering algorithm in the Scikit-learn machine learning library for cluster analysis. Select appropriate clustering parameters, such as the number of clusters, distance metrics, etc., according to the characteristics of cultural and tourism data and business needs. Execute the clustering algorithm, obtain clustering results, and evaluate their effectiveness. Construct user portrait labels and feature descriptions based on clustering results and user behavior data. Apply the constructed user portrait to the tourism recommendation system, and recommend suitable tourism products to the user based on the user's portrait label. Use user portraits for precision marketing and service optimization to improve user satisfaction and platform revenue.

[0084] The multimodal interaction module involved in this embodiment has an interactive interface unit, a voice recognition unit, an image recognition unit, and a gesture control and interaction unit to meet various interactive requirements. It can also achieve seamless switching and collaborative work between different modes. At the same time, users can also use gestures to zoom in, zoom out, or rotate the picture to obtain more detailed information.

[0085] like Figure 3 As shown, the present application also provides an electronic device, including a display module 103, a memory 102, a processor 101, and a computer program stored in the memory and executable on the processor 101, wherein the processor 101 implements the steps of a method for parsing and loading a GIM model of a power transmission project when executing the program.

[0086] In embodiments of the present invention, electronic devices include, but are not limited to, laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the embodiments of the present application described and / or required herein.

[0087] In the embodiment of the present application, the processor 101 can be implemented by using at least one of an application-specific integrated circuit (ASIC), a programmable logic device (PLD), a field programmable gate array (FPGA), a processor, a controller, a microcontroller, a microprocessor, and an electronic unit designed to perform the functions described herein. In some cases, such an implementation can be implemented in a controller. For software implementation, implementations such as processes or functions can be implemented with separate software modules that allow execution of at least one function or operation. The software code can be implemented by a software application (or program) written in any appropriate programming language, and the software code can be stored in a memory and executed by a controller.

[0088] The display module 103 is used to display information input by the user or information provided to the user. The display module 103 may include a display panel, which may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), etc.

[0089] The memory 102 may be used to store software programs and various data. The memory 102 may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0090] The present application also provides a storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the multimodal interactive tourism recommendation method are implemented.

[0091] The storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0092] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multimodal interactive tourism recommendation method, characterized in that: Methods include: Construct user travel profiles based on clustering and anomaly detection; Construct user portrait labels and feature descriptions based on clustering results and user behavior data, and configure the constructed user portraits into the multimodal interaction module; Based on the multimodal interaction module, an interactive interface unit, a speech recognition unit, an image recognition unit, and a gesture control and interaction unit are configured; The multimodal interaction module generates a VR panorama based on the voice information, image information and gestures input by the user, and provides voice and tactile feedback to the interaction interface unit.

2. The multimodal interactive tourism recommendation method according to claim 1, characterized in that: The steps to build a user travel profile based on clustering and anomaly detection also include: Obtain the user's historical travel data, which includes the user's travel destination, stay time, consumption information, and travel trajectory information; Preprocessing of historical tourism data; Group historical travel data based on clustering algorithms to identify travel behavior patterns corresponding to different user groups; Integrate the cluster analysis results to construct user travel portraits.

3. The multimodal interactive tourism recommendation method according to claim 2, characterized in that: In the method, after obtaining the clustering results, the anomaly detection algorithm is used to identify the historical tourism data for detection and obtain abnormal behaviors.

4. The multimodal interactive tourism recommendation method according to claim 2, characterized in that: The steps to configure the constructed user portrait into the multimodal interaction module also include: Extract the user's historical travel data and convert it into a numerical vector; Configure the extraction of user travel key features, define user travel portrait tags based on the extracted user travel key features, and describe the features of each user travel portrait tag; Configure the user's key travel features into the travel application scenario of the multimodal interaction module, and configure the interaction method based on the user's travel portrait tags; Among them, the configuration item PZ of the user's key tourism features in the tourism application scenario of the multimodal interaction module is: PZ = σ(QxL yi + QyLyi−1 + pz); Among them, yi is the extracted historical travel data of the i-th user, QxL, QyL are weight matrices, pz is the bias term, and σ is the activation function.

5. The multimodal interactive tourism recommendation method according to claim 1, characterized in that: The method also includes, based on the user's travel profile, combining the following travel recommendation algorithm, recommending travel data to the user; User recommended travel data Sv = f(L1, L2, L3, Ln); f() represents the feature vector fusion function, Ln represents the user's key travel features, and n represents the number of user's key travel features; f() is used to fuse two or more user travel key features into one user travel key feature.

6. The multimodal interactive tourism recommendation method according to claim 1, characterized in that: Gesture control methods include: Use VR glove sensors to collect user hand motion data; Extracting gesture features from user hand motion data; Classify and recognize gesture features using preset algorithms; The recognized gesture actions are mapped to a predefined instruction set to realize the association between gestures and operations, and an interactive tourism interface corresponding to the gesture actions is displayed in the interactive interface unit.

7. The multimodal interactive tourism recommendation method according to claim 6, characterized in that: The preset algorithm in the step of using the preset algorithm to classify and recognize gesture features is: Interactive tourism interface Mc=∑p∑qT(r+p,j+q)H(p,q); The interactive tourism interface Mc is calculated by sliding the convolution kernel H on the input image T; r and j represent the row index and column index on the user's travel key features; r and j traverse all user travel key features; p and q represent the row index and column index of the convolution kernel H respectively; In the process of gesture feature recognition, p and q will traverse all elements of the convolution kernel; H(p,q) represents the element value of the convolution kernel H in the pth row and qth column; the convolution kernel is a weight matrix used to extract the features of the user's travel portrait; T(r+p,j+q) represents the element value of the user's travel key feature at the position corresponding to the convolution kernel H; When the convolution kernel slides on the user's key travel features, T(r+p,j+q) points to different positions on the user's key travel features accordingly; by multiplying the convolution kernel with the corresponding elements of the input image and summing them, the interactive travel interface Mc is obtained.

8. A multimodal interactive tourism recommendation system, characterized in that: The system is used to implement the multimodal interactive tourism recommendation method as described in any one of claims 1 to 7; The system includes: The portrait construction module is used to construct user travel portraits based on the combination of clustering and anomaly detection; The portrait configuration module is used to construct user portrait labels and feature descriptions based on clustering results and user behavior data, and configure the constructed user portraits into the multimodal interaction module; The multimodal interaction module is configured with an interaction interface unit, a speech recognition unit, an image recognition unit, and a gesture control and interaction unit; The multimodal interaction module generates a VR panorama based on the voice information, image information and gestures input by the user, and provides voice and tactile feedback to the interaction interface unit.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the multimodal interactive tourism recommendation method as described in any one of claims 1 to 7 are implemented.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the multimodal interactive tourism recommendation method as described in any one of claims 1 to 7 are implemented.

Images