Travel site selection method and system based on multi-modal fusion

Abstract

The invention relates to a travel site selection method based on multi-modal fusion, and the method comprises the steps: obtaining the site scale data, the popularity POI data and the flow time sequence data of candidate point locations, and carrying out the location selection of the candidate point locations through a pre-constructed multi-modal fusion deep neural network model, and feature extraction and feature analysis are carried out on the place scale data, the popularity POI data and the flow time sequence data to obtain scale features, POI features, time sequence features and a cross-modal feature interaction relationship, the scale features, the POI features and the time sequence features are fused according to the cross-modal feature interaction relationship, and recommendation values of candidate point locations are generated based on a fusion result. Through the method and the device, the problem of low accuracy of travel site selection decision is solved, multiple modal data are considered, one-sided decision basis caused by a single data type is avoided, the deep neural network is adopted to fuse the multi-modal data and analyze deep association among the data, and the accuracy of site selection decision is improved.

Description

Technical Field

[0001] The present application relates to the field of big data analysis, and in particular to a travel site selection method and system based on multimodal fusion. Background Art

[0002] With the acceleration of urbanization and the diversification of travel needs, rational planning of the site selection of travel service points has become an important part of improving user experience and optimizing resource allocation.

[0003] Existing site selection systems usually rely on a single data source or use simple statistical models. A single data source usually cannot provide sufficiently diverse information and cannot fully capture various potential variables, resulting in one-sided decision-making. Simple statistical models usually cannot capture the relationship between data, resulting in low accuracy and reliability of site selection decisions.

[0004] Currently, no effective solution has been proposed to address the problem of low accuracy of travel location decisions in related technologies. Summary of the invention

[0005] The embodiments of the present application provide a travel site selection method, system, electronic device and storage medium based on multimodal fusion to at least solve the problem of low accuracy of travel site selection decisions in related technologies.

[0006] In a first aspect, an embodiment of the present application provides a travel site selection method based on multimodal fusion, the method comprising: Obtain site size data, popular POI data, and traffic time series data of candidate locations; Through the pre-built multi-modal fusion deep neural network model, feature extraction and feature analysis are performed on the venue scale data, the popular POI data and the traffic time series data respectively to obtain scale features, POI features and time series features, as well as cross-modal feature interaction relationships; Through the multimodal fusion deep neural network model, according to the cross-modal feature interaction relationship, the scale feature, the POI feature and the time series feature are fused, and the recommendation value of the candidate point is generated based on the fusion result.

[0007] In some of the embodiments, the multimodal fusion deep neural network model includes a data preprocessing layer, a feature fusion layer and a decision layer; The data preprocessing layer performs feature extraction on the venue scale data, the popular POI data and the traffic time series data to obtain scale features, POI features and time series features, and respectively obtains feature interaction relationships among the scale features, the POI features and the time series features; The feature fusion layer obtains a cross-modal feature interaction relationship according to the feature interaction relationship among the scale feature, the POI feature and the time series feature, and fuses the scale feature, the POI feature and the time series feature based on the cross-modal feature interaction relationship to obtain a fused feature; The decision layer generates a recommendation value for the candidate point based on the fusion feature.

[0008] In some embodiments, obtaining the feature interaction relationship of the scale feature includes: The scale feature is screened by a gradient boosting tree to obtain first feature data; The first feature data is input into a multi-layer perceptron to obtain the feature interaction relationship of the scale feature.

[0009] In some embodiments, obtaining the feature interaction relationship of the POI feature includes: Performing spatial relationship modeling on the POI features through a graph neural network to obtain a spatial relationship model; Embed the node structure information of the spatial relationship model into the vector representation of the POI feature to obtain second feature data; The second feature data is input into a multi-layer perceptron to obtain the feature interaction relationship of the POI features.

[0010] In some embodiments, the acquiring of the feature interaction relationship of the time series feature includes: Through the Transformer network, a global time dependency relationship of the time series features is constructed; By using a long short-term memory network, long-term trends are captured according to the global time dependency; Based on the long-term trend, the feature interaction relationship of the time series features is obtained.

[0011] In some embodiments, the method further comprises: Constructing the multimodal fusion deep neural network model based on the binary cross entropy loss function; The multimodal fusion deep neural network model is trained through a data enhancement strategy, and the multimodal fusion deep neural network model is a binary classification model; The performance of the trained multimodal fusion deep neural network model was evaluated by accuracy, recall and F1 score.

[0012] In some embodiments, the method further comprises: Performing weighted calculation on the scale feature, the POI feature and the time series feature to obtain a comprehensive score for each of the candidate points; Based on the preset grading rules and the comprehensive score, determine the recommended star rating corresponding to each of the candidate points; A point recommendation list is generated according to the recommendation star ratings and the recommendation values.

[0013] In some embodiments, before extracting features from the venue scale data, the popular POI data, and the traffic time series data, the method further includes: By forward filling and backward filling, missing values ​​of the traffic time series data are supplemented; Supplement the missing values ​​of the venue scale data and the popular POI data by filling in the mean or the category mode; Based on the quantile detection method, extreme values ​​in the venue scale data, the popular POI data and the traffic time series data are eliminated.

[0014] In a second aspect, an embodiment of the present application provides a travel site selection system based on multimodal fusion, the system comprising: The data acquisition module is used to obtain the venue scale data, popular POI data and traffic time series data of the candidate points; A feature extraction module is used to extract and analyze the features of the venue scale data, the popular POI data and the traffic time series data respectively through a pre-built multi-modal fusion deep neural network model to obtain scale features, POI features and time series features, as well as cross-modal feature interaction relationships; The recommendation module is used to fuse the scale feature, the POI feature and the time series feature through the multimodal fusion deep neural network model according to the cross-modal feature interaction relationship, and generate a recommendation value for the candidate point based on the fusion result.

[0015] In a third aspect, an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the travel site selection method based on multimodal fusion as described in the first aspect above is implemented.

[0016] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the travel site selection method based on multimodal fusion as described in the first aspect above.

[0017] Compared with the related art, the travel site selection method based on multimodal fusion provided in the embodiment of the present application obtains the site scale data, hot POI data and flow time series data of the candidate points, and extracts and analyzes the features of the site scale data, hot POI data and flow time series data respectively through the pre-built multimodal fusion deep neural network model, and obtains the scale features, POI features and time series features, as well as the cross-modal feature interaction relationship. According to the cross-modal feature interaction relationship, the scale features, POI features and time series features are fused, and the recommended value of the candidate point is generated based on the fusion result, which solves the problem of low accuracy of travel site selection decision. Comprehensively consider the site scale attributes, hot POI data, and flow time series data to avoid the one-sided decision basis caused by a single data type. At the same time, a deep neural network is used to fuse multimodal data to analyze the deep-level association between data, thereby improving the accuracy of site selection decisions. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1 is a flow chart of a travel site selection method based on multimodal fusion according to an embodiment of the present application; Figure 2 is a schematic diagram of the architecture of a multimodal fusion deep neural network model according to an embodiment of the present application; Figure 3 is a flowchart of a multimodal fusion deep neural network model training method according to an embodiment of the present application; Figure 4 is a structural block diagram of a travel location selection system based on multimodal fusion according to an embodiment of the present application; Figure 5 It is a schematic diagram of the internal structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0020] Obviously, the drawings described below are only some examples or embodiments of the present application. For ordinary technicians in this field, the present application can also be applied to other similar scenarios based on these drawings without creative work. In addition, it can also be understood that although the efforts made in this development process may be complicated and lengthy, for ordinary technicians in this field related to the content disclosed in this application, some changes in design, manufacturing or production based on the technical content disclosed in this application are just conventional technical means, and should not be understood as insufficient content disclosed in this application.

[0021] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0022] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantity limitation, and may indicate the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0023] This embodiment provides a travel location selection method based on multimodal fusion. Figure 2 is a flow chart of a travel location selection method based on multimodal fusion according to an embodiment of the present application, such as Figure 2As shown, the process includes the following steps: Step S101, obtaining site scale data, popular POI data and traffic time series data of candidate points.

[0024] Optionally, venue scale data is collected through government public data, including but not limited to venue area, functional type (such as residential area, shopping mall, office area), and floor height. Popular POI data is collected from the data development platform, including but not limited to passenger flow, visit frequency, dwelling time, and distribution of public facilities, which can reflect the regional economic vitality and potential user needs. Traffic time series data is obtained through the processing of charging pile data and cabinet data on the network platform, including but not limited to the average daily number of battery replacement orders and the number of riders passing through within a preset number of days (such as 3 days).

[0025] The venue scale data reflects the type of candidate points; the popular POI data focuses on the popularity of the candidate points and their activity in a specific time period, reflecting the popularity distribution of different points of interest; the traffic time series data focuses on analyzing the changes in traffic over time, such as traffic fluctuations on a road, business district or website, to identify trends or make predictions.

[0026] By analyzing three modes of data, namely venue scale data, popular POI data and traffic time series data, we can avoid one-sided decision-making due to a single data type and improve the accuracy of the analysis results.

[0027] Step S102, using a pre-built multimodal fusion deep neural network model, feature extraction and feature analysis are performed on the venue scale data, popular POI data, and traffic time series data to obtain scale features, POI features, time series features, and cross-modal feature interaction relationships; The collected venue scale data, popular POI data and traffic time series data are processed and feature extracted. In terms of feature processing, the input data is standardized and the value range is compressed to the [0,1] interval to reduce the scale difference between features and avoid affecting the model learning effect.

[0028] Step S103, through the multimodal fusion deep neural network model, according to the cross-modal feature interaction relationship, the scale feature, the POI feature and the time series feature are fused, and the recommendation value of the candidate point is generated based on the fusion result.

[0029] In some of these embodiments, the multimodal fusion deep neural network model includes a data preprocessing layer, a feature fusion layer, and a decision layer; The data preprocessing layer extracts features from the venue scale data, popular POI data, and traffic time series data to obtain scale features, POI features, and time series features, and obtains the feature interaction relationships among scale features, POI features, and time series features.

[0030] The feature fusion layer obtains the cross-modal feature interaction relationship according to the feature interaction relationship among the scale feature, POI feature and time series feature, and fuses the scale feature, POI feature and time series feature based on the cross-modal feature interaction relationship to obtain the fused feature.

[0031] The decision layer generates recommended values ​​for candidate points based on fused features.

[0032] Figure 2 is a schematic diagram of the architecture of a multimodal fusion deep neural network model according to an embodiment of the present application, such as Figure 2 As shown, the multimodal fusion deep neural network model includes a data preprocessing layer, a feature fusion layer, a decision layer and a prediction output layer.

[0033] This embodiment improves the deep neural network, extracts and fuses multimodal data, fully explores the deep-level correlations between data, and improves the accuracy of site selection prediction.

[0034] In some embodiments, obtaining the feature interaction relationship of the scale feature includes: Step S201, screen the scale feature through the gradient boosting tree to obtain the first feature data.

[0035] Step S202: input the first feature data into a multi-layer perceptron to obtain a feature interaction relationship of the scale feature.

[0036] Preferably, XGBoost is used to screen out key features to reduce redundancy, and then the streamlined feature vectors are input into the MLP network to model nonlinear feature interactions.

[0037] In some embodiments, obtaining the feature interaction relationship of the POI features includes: Step S301: spatial relationship modeling of POI features is performed through a graph neural network to obtain a spatial relationship model.

[0038] Step S302: embed the node structure information of the spatial relationship model into the vector representation of the POI feature to obtain second feature data.

[0039] Step S303: input the second feature data into a multi-layer perceptron to obtain the feature interaction relationship of the POI features.

[0040] Preferably, the spatial relationship of POI features is modeled through a graph neural network (GNN), the node structure information is embedded into the vector representation, and the complex feature patterns are further learned through a multi-layer perceptron (MLP) to obtain the feature interaction relationship of POI features.

[0041] In some embodiments, obtaining the feature interaction relationship of the time series features includes: Step S401, constructing the global time dependency of the time series features through the Transformer network.

[0042] Step S402, capturing long-term trends based on global time dependencies through a long short-term memory network.

[0043] Step S403, based on the long-term trend, obtain the feature interaction relationship of the time series features.

[0044] In this embodiment, the global time dependency is modeled by the Transformer network, and then the long short-term memory network (LSTM) is used to capture the long-term trend to obtain a complete time series representation.

[0045] After obtaining the cross-modal feature interaction relationship of scale features, POI features, and time series features, the feature fusion layer uses the Transformer network to model the interaction relationship between cross-modal features, and through the self-attention mechanism, selectively focuses on the relevant features between different modalities, while suppressing irrelevant noise features. The fused feature vector is passed to the decision layer, and the binary classification task is completed through MLP, and the recommended value of the candidate point is output. Preferably, the recommended value includes 0 and 1, where 0 means not recommended and 1 means recommended.

[0046] The design of the model fully considers the heterogeneity and correlation of the data to ensure the accuracy and reliability of the final output results.

[0047] Through the above steps S101 to S103, the venue scale data, hot POI data and flow time series data of the candidate points are obtained. Through the pre-built multimodal fusion deep neural network model, the venue scale data, hot POI data and flow time series data are respectively extracted and analyzed to obtain scale features, POI features and time series features, as well as cross-modal feature interactions. According to the cross-modal feature interaction, the scale features, POI features and time series features are fused, and the recommended value of the candidate points is generated based on the fusion results, which solves the problem of low accuracy of travel site selection decisions. Comprehensively consider the venue scale attributes, hot POI data, and flow time series data to avoid the one-sided decision-making basis caused by a single data type. At the same time, a deep neural network is used to fuse multimodal data to analyze the deep-level associations between data, thereby improving the accuracy of site selection decisions.

[0048] In some embodiments, the method further comprises: Step S501, constructing a multimodal fusion deep neural network model based on a binary cross entropy loss function.

[0049] Step S502, training the multimodal fusion deep neural network model through a data enhancement strategy, where the multimodal fusion deep neural network model is a binary classification model; Step S503, evaluating the performance of the trained multimodal fusion deep neural network model through accuracy, recall and F1 score.

[0050] The model uses the binary cross entropy loss function (BCE) to measure the difference between the predicted value and the true label, which is expressed as:

[0051] Among them, y i Represents the sample category, whose value is 0 or 1; p i Represents the probability of predicting the sample category. The ultimate goal of the model is to minimize the loss function.

[0052] Optionally, the model optimizer uses the Adam algorithm, and the learning rate is set to 0.0002 to ensure the stability of gradient updates and the speed of training convergence. The training data set is divided into a training set and a validation set at an 8:2 ratio. During the training process, the accuracy of the validation set is monitored, and the model parameters are adjusted dynamically. In order to improve the generalization ability of the model, data enhancement strategies are used, including but not limited to randomly discarding some features and adding Gaussian noise to time series features. The number of model training rounds is set to 1000 times, and the model with the best performance in the validation set is saved as the final output during the training process. The model performance is evaluated by indicators such as accuracy, recall rate, and F1 score to ensure that it has good predictive ability and generalizability. Figure 3 It is a flowchart of a multimodal fusion deep neural network model training method according to an embodiment of the present application.

[0053] When the model is deployed as a service, the trained model weights are encapsulated through the microservice framework Flask. The service receives the input request address. For each point, the network predicts the probability value of its suitability for setting up the target service (such as a car rental point) and determines the final result (0 or 1) through the threshold.

[0054] After the model is deployed, operational data can be collected through a continuous dynamic feedback mechanism, and the actual results can be used as input for a new round of training data to gradually improve the model's prediction accuracy and adaptability.

[0055] In some embodiments, the method further comprises: Step S601, weighted calculation is performed on the scale feature, POI feature and time series feature to obtain a comprehensive score for each candidate point.

[0056] Step S602: Based on the preset grading rules and the comprehensive score, determine the recommendation star rating corresponding to each candidate point.

[0057] Step S603: Generate a list of recommended spots based on the recommended star ratings and recommended values.

[0058] In order to further improve the quality of decision-making, the comprehensive score is calculated in combination with the prediction results. The scoring formula is the weighted sum of the scores of each modal feature. Optionally, the comprehensive score range is [0, 100]. The points are graded according to the score. 90-100 is a 3-star priority recommendation point, 75-90 is a 2-star second priority recommendation point, 60-75 is a 1-star ordinary recommendation point, and points below 60 are not recommended. Finally, combined with the recommendation list output by the system, the operation and maintenance is assisted to optimize resource allocation and achieve accurate site selection.

[0059] In some embodiments, before extracting features from the venue scale data, the popular POI data, and the traffic time series data, the method further includes: Step S701, supplementing the missing values ​​of the traffic time series data by forward filling and backward filling; Step S702, supplementing the missing values ​​of the venue scale data and the popular POI data by filling in the mean value or the category mode; Step S703: based on the quantile detection method, remove extreme values ​​in the venue scale data, popular POI data and traffic time series data.

[0060] After obtaining the site scale data, popular POI data and traffic time series data of the candidate points, these data need to be processed to ensure the quality and consistency of the input data.

[0061] Processing of missing values: For time series data, a combination of forward filling and backward filling is used to ensure the integrity of key time series features; for non-time series data, mean filling or category mode filling is used.

[0062] Detect and correct outliers: Use quantile-based detection methods to remove extreme values ​​to improve the robustness of the model.

[0063] After cleaning, all data are formatted uniformly.

[0064] The above method uses multimodal data fusion and deep neural network models to integrate venue scale attributes, popular POI data and traffic time series data to achieve accurate and efficient travel service point layout. The multimodal fusion deep neural network model can not only improve the scientificity and accuracy of site selection prediction and optimize resource allocation, but also reduce the operating costs of enterprises while taking into account the user experience.

[0065] It should be noted that the steps shown in the above process or the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0066] This embodiment also provides a travel site selection system based on multimodal fusion, which is used to implement the above embodiments and preferred implementation modes, and will not be repeated here. As used below, the terms "module", "unit", "subunit", etc. can implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0067] Figure 4 is a structural block diagram of a travel location selection system based on multimodal fusion according to an embodiment of the present application, such as Figure 4 As shown, the system includes: The data acquisition module 41 is used to obtain the site scale data, popular POI data and traffic time series data of the candidate points.

[0068] The feature extraction module 42 is used to extract and analyze the features of the venue scale data, popular POI data and traffic time series data respectively through a pre-built multimodal fusion deep neural network model to obtain scale features, POI features and time series features, as well as cross-modal feature interaction relationships.

[0069] The recommendation module 43 is used to generate a recommendation value of the candidate point based on the fusion result by fusing the scale feature, POI feature and time series feature through a multimodal fusion deep neural network model according to the cross-modal feature interaction relationship.

[0070] In some of the embodiments, the multimodal fusion deep neural network model includes a data preprocessing layer, a feature fusion layer and a decision layer.

[0071] The data preprocessing layer extracts features from the venue scale data, popular POI data, and traffic time series data to obtain scale features, POI features, and time series features, and obtains the feature interaction relationships among scale features, POI features, and time series features.

[0072] The feature fusion layer obtains the cross-modal feature interaction relationship according to the feature interaction relationship among the scale feature, POI feature and time series feature, and fuses the scale feature, POI feature and time series feature based on the cross-modal feature interaction relationship to obtain the fused feature.

[0073] The decision layer generates recommended values ​​for candidate points based on fused features.

[0074] In some embodiments, the data preprocessing layer includes: a first preprocessing module, which is used to screen the scale features through a gradient boosting tree to obtain first feature data, input the first feature data into a multi-layer perceptron, and obtain the feature interaction relationship of the scale features.

[0075] In some of the embodiments, the data preprocessing layer includes: a second preprocessing module, which is used to perform spatial relationship modeling on POI features through a graph neural network to obtain a spatial relationship model; embed the node structure information of the spatial relationship model into the vector representation of the POI features to obtain second feature data; and input the second feature data into a multi-layer perceptron to obtain the feature interaction relationship of the POI features.

[0076] In some of the embodiments, the data preprocessing layer includes: a third preprocessing module, which is used to construct a global time dependency of time series features through a Transformer network; to capture long-term trends based on the global time dependency through a long short-term memory network; and to obtain feature interaction relationships of time series features based on long-term trends.

[0077] In some of these embodiments, the system further includes: a model building module, a model training module and a model evaluation module.

[0078] Model building module, used to build a multimodal fusion deep neural network model based on binary cross entropy loss function; The model training module is used to train the multimodal fusion deep neural network model through data enhancement strategy. The multimodal fusion deep neural network model is a binary classification model; The model evaluation module is used to evaluate the performance of the trained multimodal fusion deep neural network model through accuracy, recall and F1 score.

[0079] In some embodiments, the system further comprises: The scoring module is used to perform weighted calculation on scale features, POI features and time series features to obtain a comprehensive score for each candidate point.

[0080] The recommendation list generation module is used to determine the recommendation star rating corresponding to each candidate point based on the preset grading rules and the comprehensive score, and generate a point recommendation list according to each recommendation star rating and recommendation value.

[0081] In some embodiments, the data acquisition module further includes: The first data correction module is used to supplement the missing values ​​of the traffic time series data through forward filling and backward filling.

[0082] The second data correction module is used to supplement the missing values ​​of the venue scale data and the popular POI data by filling in the mean or the category mode.

[0083] The third data correction module is used to eliminate extreme values ​​in venue scale data, popular POI data and traffic time series data based on the quantile detection method.

[0084] Through the above system, the problem of low accuracy of travel site selection decisions has been solved. The scale attributes of the venue, popular POI data, and traffic time series data are comprehensively considered to avoid one-sided decision-making due to a single data type. At the same time, deep neural networks are used to fuse multimodal data and analyze the deep correlation between data, thereby improving the accuracy of site selection decisions.

[0085] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0086] This embodiment further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0087] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0088] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program: S1, obtain the venue scale data, popular POI data and traffic time series data of the candidate points.

[0089] S2, through the pre-built multimodal fusion deep neural network model, feature extraction and feature analysis are performed on the venue scale data, popular POI data and traffic time series data respectively to obtain scale features, POI features and time series features, as well as cross-modal feature interactions.

[0090] S3, through the multimodal fusion deep neural network model, according to the cross-modal feature interaction relationship, integrates scale features, POI features and time series features, and generates recommended values ​​for candidate points based on the fusion results.

[0091] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0092] In one embodiment, Figure 5 is a schematic diagram of the internal structure of an electronic device according to an embodiment of the present application, such as Figure 5 As shown, an electronic device is provided, which may be a server, and its internal structure diagram may be as shown in Figure 5As shown. The electronic device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the electronic device is used to store data. The network interface of the electronic device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a travel site selection method based on multimodal fusion is implemented.

[0093] Those skilled in the art will understand that Figure 5 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0094] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0095] Those skilled in the art should understand that the technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0096] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A travel location selection method based on multimodal fusion, characterized in that: The method comprises: Obtain site size data, popular POI data, and traffic time series data of candidate locations; Through the pre-built multi-modal fusion deep neural network model, feature extraction and feature analysis are performed on the venue scale data, the popular POI data and the traffic time series data respectively to obtain scale features, POI features and time series features, as well as cross-modal feature interaction relationships; Through the multimodal fusion deep neural network model, according to the cross-modal feature interaction relationship, the scale feature, the POI feature and the time series feature are fused, and the recommendation value of the candidate point is generated based on the fusion result.

2. The method according to claim 1, characterized in that The multimodal fusion deep neural network model includes a data preprocessing layer, a feature fusion layer and a decision layer; The data preprocessing layer performs feature extraction on the venue scale data, the popular POI data and the traffic time series data to obtain scale features, POI features and time series features, and respectively obtains feature interaction relationships among the scale features, the POI features and the time series features; The feature fusion layer obtains a cross-modal feature interaction relationship according to the feature interaction relationship among the scale feature, the POI feature and the time series feature, and fuses the scale feature, the POI feature and the time series feature based on the cross-modal feature interaction relationship to obtain a fused feature; The decision layer generates a recommendation value for the candidate point based on the fusion feature.

3. The method according to claim 2, characterized in that The feature interaction relationship of obtaining the scale feature includes: The scale feature is screened by a gradient boosting tree to obtain first feature data; The first feature data is input into a multi-layer perceptron to obtain the feature interaction relationship of the scale feature.

4. The method according to claim 2, characterized in that: The feature interaction relationship of acquiring the POI feature includes: Performing spatial relationship modeling on the POI features through a graph neural network to obtain a spatial relationship model; Embed the node structure information of the spatial relationship model into the vector representation of the POI feature to obtain second feature data; The second feature data is input into a multi-layer perceptron to obtain the feature interaction relationship of the POI features.

5. The method according to claim 2, characterized in that: The acquiring of the feature interaction relationship of the time series feature comprises: Through the Transformer network, a global time dependency relationship of the time series features is constructed; By using a long short-term memory network, long-term trends are captured according to the global time dependency; Based on the long-term trend, the feature interaction relationship of the time series features is obtained.

6. The method according to claim 1, characterized in that The method further comprises: Constructing the multimodal fusion deep neural network model based on the binary cross entropy loss function; The multimodal fusion deep neural network model is trained through a data enhancement strategy, and the multimodal fusion deep neural network model is a binary classification model; The performance of the trained multimodal fusion deep neural network model was evaluated by accuracy, recall and F1 score.

7. The method according to claim 1, characterized in that The method further comprises: Performing weighted calculation on the scale feature, the POI feature and the time series feature to obtain a comprehensive score for each of the candidate points; Based on the preset grading rules and the comprehensive score, determine the recommended star rating corresponding to each of the candidate points; A point recommendation list is generated according to the recommendation star ratings and the recommendation values.

8. The method according to claim 1, characterized in that Before extracting features from the venue scale data, the popular POI data, and the traffic time series data respectively, the method further includes: By forward filling and backward filling, missing values ​​of the traffic time series data are supplemented; Supplement the missing values ​​of the venue scale data and the popular POI data by filling in the mean or the category mode; Based on the quantile detection method, extreme values ​​in the venue scale data, the popular POI data and the traffic time series data are eliminated.

9. A travel location selection system based on multimodal fusion, characterized in that: The system comprises: The data acquisition module is used to obtain the venue scale data, popular POI data and traffic time series data of the candidate points; A feature extraction module is used to extract and analyze the features of the venue scale data, the popular POI data and the traffic time series data respectively through a pre-built multi-modal fusion deep neural network model to obtain scale features, POI features and time series features, as well as cross-modal feature interaction relationships; The recommendation module is used to fuse the scale feature, the POI feature and the time series feature through the multimodal fusion deep neural network model according to the cross-modal feature interaction relationship, and generate a recommendation value for the candidate point based on the fusion result.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the travel site selection method based on multimodal fusion as described in any one of claims 1 to 8 is implemented.

Images