Panoramic image multi-source information fusion system

Abstract

The invention relates to the technical field of multi-source information fusion, in particular to a panoramic image multi-source information fusion system. Comprising a front-end framework, the front-end framework runs based on a WeChat applet, and the front-end framework comprises a colored drawing map module, a panorama display module, a scenic spot detail module and a navigation interaction module; and the colored drawing map module is used for macroscopically displaying global travel resource distribution and providing a navigation entrance. According to the invention, a panoramic image is dynamically bound with characters, audios and videos through a multi-modal fusion engine, and accurate information association is realized through natural language processing and an audio timestamp synchronization technology; an improved multi-band fusion algorithm is utilized to eliminate splicing gaps, the navigation precision is improved in combination with GIS coordinate mapping and GeoHash spatial index, a rear end supports high concurrent access through load balancing and cloud server elastic expansion, and a front end integrates a VR mode and a collaborative filtering recommendation algorithm to optimize immersive experience.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-source information fusion, and in particular to a panoramic image multi-source information fusion system. Background Art

[0002] Panoramic images are a form of visual presentation covering 360-degree scenes generated through image stitching technology or dedicated panoramic cameras. They can simulate the real environment observed by the human eye and provide an immersive experience. Their core features include a large field of view, multi-source data integration, and interactive browsing functions. They are widely used in virtual tourism, digital mapping, and cultural heritage protection.

[0003] Generally, traditional panoramic image data fusion methods mainly rely on single sensors or manual stitching techniques, such as image alignment based on feature point matching and simple linear fusion algorithms. However, these methods have significant shortcomings. First, multi-source heterogeneous data are difficult to align and synchronize efficiently, resulting in information fragmentation. Second, stitching in dynamic scenes is prone to "ghosting" or breakage, and lacks the ability to process spatiotemporal consistency. Traditional methods have high computational complexity when processing high-resolution panoramic images, making it difficult to respond in real time. In addition, they lack deep fusion of multimodal information, which limits the integrity of scene understanding and the immersiveness of user experience.

[0004] Based on this, the present invention provides a panoramic image multi-source information fusion system to solve the above-mentioned technical problems. Summary of the Invention

[0005] The purpose of the present invention is to provide a panoramic image multi-source information fusion system to solve the problems raised by the above background technology.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] The present invention proposes a panoramic image multi-source information fusion system, comprising:

[0008] A front-end framework, which runs on the WeChat applet and includes a painted map module, a panoramic display module, a scenic spot details module, and a navigation interaction module;

[0009] The painted map module is used to macroscopically display the distribution of cultural and tourism resources in the entire region and provide a navigation portal;

[0010] The panoramic display module is used to display panoramic images of a single scenic spot at the meso-level, supporting 360° interactive browsing;

[0011] The attraction details module is used to display text, pictures, audio, and video multi-source information of specific attractions at a micro level;

[0012] The navigation interaction module is used to provide users with path planning, search and immersive experience functions;

[0013] It is characterized by further comprising:

[0014] Backend framework, which includes information collection and processing module, panoramic stitching and fusion module, map data management module, service interface module and operation and maintenance deployment module;

[0015] The information acquisition and processing module is used to integrate panoramic images, text, audio, and video multi-source data to complete standardization processing;

[0016] The panoramic stitching and fusion module is used to seamlessly stitch panoramic pictures taken from multiple angles and fuse other media information;

[0017] The map data management module is used to manage the painted map data, scenic spot coordinates and related information;

[0018] The service interface module is used to provide an API interface called by the front end to support real-time response of data interaction;

[0019] The operation and maintenance deployment module is used to ensure stable operation of the system and support high concurrent access.

[0020] Preferably, the painted map module is further used to provide an area information bar, a point selection bar and a flashing hotspot;

[0021] The regional information bar divides administrative division boundaries based on GIS geocoding technology and displays the distribution of cultural and tourism resources in each administrative district, allowing users to quickly locate the target area;

[0022] The point selection bar uses the Vue / React framework to dynamically render the list, combines JSON data to bind the tour route sequence, and arranges the entrances of scenic spots according to the preset optimal tour route to guide users to browse the global scenic spots in order;

[0023] The flashing hotspot uses Canvas or WebGL to draw interactive hot zones, and realizes dynamic annotation through JavaScript event monitoring, marking the name of the scenic spot in the high-altitude bird's-eye view panoramic map, so that the user can click to jump to the corresponding scenic spot panoramic interface.

[0024] Preferably, the panoramic display module is further used to provide a panoramic image player, scene switching buttons and commentary for synchronous display;

[0025] The panoramic image player is based on the Three.js or Marzipano panoramic rendering library, supports WebGL hardware acceleration, loads and renders 360° panoramic images, and allows users to freely explore the scene by dragging, dropping, and zooming.

[0026] The scene switching button is linked to the panoramic SDK interface through the front-end routing to trigger the perspective switch and jump between different observation points. The observation points include the front elevation and the high-altitude view, which are used to display the full view of the scenic spot from multiple angles.

[0027] The synchronous display of the commentary adopts the WebVTT subtitle format and is bound to the timeline of the panoramic player to achieve dynamic text matching. The corresponding commentary is automatically displayed as the panoramic viewing angle changes, which is used to enhance the timeliness of scene information transmission.

[0028] Preferably, the scenic spot details module is further used to provide a graphic information card, an audio player and a video embed window;

[0029] The graphic information card uses HTML / CSS structured layout and combines AJAX to dynamically load Markdown or rich text data to display the historical background and cultural value description of the scenic spot, providing in-depth cultural interpretation;

[0030] The audio player is based on the HTML5 Audio tag or the Howler.js audio library, supports MP3 / WAV format streaming media playback, and plays professionally recorded scenic spot commentary dubbing to assist users in auditory immersion experience;

[0031] The video embedding window integrates Video.js or H5 native Video components, supports HLS / MPEG-DASH adaptive streaming, and embeds scenic spot promotional videos or documentaries to dynamically display the real scenery and activity scenes of the scenic spot.

[0032] Preferably, the navigation interaction module is further used to provide travel route recommendations, a search box, and a VR mode switching button;

[0033] The travel route recommendation generates a recommendation list based on tag classification and collaborative filtering algorithm, and displays customized travel routes by theme to meet the user's personalized travel needs;

[0034] The search box uses the Elasticsearch full-text search engine to implement keyword fuzzy matching and word segmentation queries, allowing users to enter scenic spot names or keywords to quickly locate target content for information retrieval;

[0035] The VR mode switch button calls WebXRAPI or CardboardSDK to switch to full-screen stereo rendering mode, enabling virtual reality helmet-compatible display to provide a fully immersive scene browsing experience.

[0036] Preferably, the information acquisition and processing module further includes a panoramic image acquisition unit and a multi-source data cleaning unit;

[0037] The panoramic image acquisition unit is used to acquire panoramic images through drones and fixed-point shooting;

[0038] The multi-source data cleaning unit is used to unify the format of multi-source data and remove redundancy.

[0039] Preferably, the panoramic stitching and fusion module further includes an image alignment unit, a seamless stitching unit and a multimodal fusion engine;

[0040] The image alignment unit extracts key feature points of images from different perspectives by matching SIFT, ORB or SURF feature points to calculate the geometric transformation matrix, which is used to align panoramic images taken from multiple angles, eliminate perspective differences, and provide a geometric consistency basis for seamless stitching;

[0041] The seamless stitching unit uses an image stitching algorithm based on OpenCV or PanoTools library to smoothly stitch the aligned images through multi-band fusion or exposure compensation, eliminating color differences and distortion at the seams to generate a complete point-to-point panoramic image.

[0042] The multimodal fusion engine uses natural language processing and audio timestamp synchronization technology to bind text commentary, background music and panoramic images, so as to achieve dynamic association between text / audio information and panoramic scenes, enhance the immersive experience, and trigger voice explanations by clicking on hotspots.

[0043] Preferably, the map data management module further comprises a map drawing engine, a database and a spatial index unit;

[0044] The mapping engine uses ArcGIS or QGIS’s coordinate mapping algorithm to map the latitude and longitude coordinates of scenic spots to the pixel positions of the painted map, ensuring the consistency between the painted map and the real geographic space and supporting accurate labeling and navigation of hotspots.

[0045] The database uses MySQL to store the name, coordinates, and description structured metadata of scenic spots, and MongoDB to store panoramic image paths and unstructured audio file data, which is used to efficiently manage multi-source heterogeneous data and support fast query and expansion;

[0046] The spatial index unit constructs a spatial index based on the R-tree or GeoHash algorithm, and combines it with the quadtree to optimize geographic range queries, so as to quickly locate hotspots around the user or in a specified area, thereby improving navigation response speed.

[0047] Preferably, the service interface module and operation and maintenance deployment module further include RESTfulAPI, WebSocket and WeChat applet SDK integration units;

[0048] The RESTful API is based on a standardized interface designed with SpringBoot or Flask frameworks, uses JSON format to transmit data, and is used to provide core services such as scenic spot information query, panorama loading, and user behavior recording.

[0049] The WebSocket establishes a two-way communication channel through one of the Socket.IO or native WebSocket protocols for real-time push navigation path updates and user location synchronization dynamic information;

[0050] The WeChat mini program SDK integration unit calls the WeChat official API to integrate login authorization, map components, and payment functions to achieve seamless docking between the mini program end and the backend service, and support one-click sharing and social dissemination.

[0051] Preferably, the operation and maintenance deployment module further includes a load balancing unit, a cloud server and a log monitoring unit;

[0052] The load balancing unit configures polling or weighted least connection algorithm through Nginx reverse proxy to distribute user requests to multiple servers to avoid single point of failure and improve system stability and throughput in high concurrency scenarios;

[0053] The cloud server deploys backend services based on Alibaba Cloud ECS or Tencent Cloud CVM, and combines object storage to host large panoramic files, providing elastic computing resources and high-availability storage, and supporting dynamic expansion and contraction;

[0054] The log monitoring unit collects log data through Elasticsearch and combines it with Kibana to visualize the response time and error rate indicators of the monitoring interface to diagnose system anomalies in real time, optimize performance bottlenecks, and ensure service reliability.

[0055] Compared with the prior art, the present invention has the following beneficial effects:

[0056] The present invention dynamically binds panoramic images with text, audio, and video through a multimodal fusion engine, and achieves precise information association through natural language processing and audio timestamp synchronization technology; utilizes an improved multi-band fusion algorithm to eliminate splicing gaps, combines GIS coordinate mapping with GeoHash spatial indexing to improve navigation accuracy, supports high-concurrency access through load balancing and elastic expansion of cloud servers on the back end, and integrates VR mode and collaborative filtering recommendation algorithm on the front end to optimize the immersive experience. The present invention realizes full-process automation from data collection and cleaning to real-time interaction, significantly improving the efficiency and visualization quality of the digitization of cultural and tourism resources, and providing an efficient, stable and user-friendly solution for cultural communication and virtual tourism. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1This is a topological diagram of the panoramic image multi-source information fusion system of the present invention;

[0058] Figure 2 This is a flowchart of the front-end framework implementation of the panoramic image multi-source information fusion system of the present invention;

[0059] Figure 3 This is a flowchart of the front-end framework implementation of the panoramic image multi-source information fusion system of the present invention;

[0060] Figure 4 This is a flow chart of the panoramic image multi-source information fusion method of the present invention. DETAILED DESCRIPTION

[0061] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0062] Example 1, please refer to Figure 1 ,The present invention proposes a panoramic image multi-source information fusion system, which is implemented based on the front-end framework and the back-end framework;

[0063] It should be noted that the front-end framework runs on the WeChat applet. Furthermore, the front-end framework includes a painted map module, a panoramic display module, a scenic spot details module, and a navigation interaction module. The back-end framework includes an information collection and processing module, a panoramic stitching and fusion module, a map data management module, a service interface module, and an operation and maintenance deployment module.

[0064] In this system, the front-end framework also needs to explain that the painted map module is used to macro-display the distribution of cultural and tourism resources in the entire region and provide a navigation portal. The panoramic display module is used to display panoramic images of a single scenic spot at the meso level and supports 360° interactive browsing. The scenic spot details module is used to display text, pictures, audio, and video multi-source information of specific scenic spots at the micro level. The navigation interaction module is used to provide users with path planning, search, and immersive experience functions.

[0065] In this system, the back-end framework also needs to explain that the information collection and processing module is used to integrate panoramic images, text, audio, and video multi-source data to complete standardized processing; the panoramic stitching and fusion module is used to seamlessly stitch panoramic pictures taken from multiple angles and integrate other media information; the map data management module is used to manage painted map data, scenic spot coordinates and related information; the service interface module is used to provide an API interface called by the front end to support real-time response to data interaction; the operation and maintenance deployment module is used to ensure stable operation of the system and support high concurrent access.

[0066] In this embodiment, it should also be noted that the painted map module is also used to provide an area information bar, a point selection bar, and a flashing hotspot;

[0067] Furthermore, the regional information bar divides administrative division boundaries based on GIS geocoding technology, showing the distribution of cultural and tourism resources in each administrative district, allowing users to quickly locate the target area;

[0068] Furthermore, the point selection bar uses the Vue / React framework to dynamically render a list, combined with JSON data binding to the tour route sequence, to arrange the entrances of scenic spots according to the preset optimal tour route, to guide users to browse the global scenic spots in order;

[0069] Furthermore, Flashing Hotspot uses Canvas or WebGL to draw interactive hot zones, and implements dynamic annotation through JavaScript event monitoring. It annotates the names of scenic spots in the high-altitude panoramic view, allowing users to click to jump to the corresponding panoramic interface of the scenic spot.

[0070] In this embodiment, it should also be noted that the panoramic display module is also used to provide a panoramic image player, scene switching buttons and commentary synchronous display;

[0071] Furthermore, the panoramic image player is based on the Three.js or Marzipano panoramic rendering library, supports WebGL hardware acceleration, loads and renders 360° panoramic images, and allows users to freely explore the scene by dragging, dropping, and zooming.

[0072] Furthermore, the scene switching button is linked to the panoramic SDK interface through the front-end router to trigger the perspective switch and jump between different observation points, including the front elevation and high-altitude view, to show the full view of the scenic spot from multiple angles;

[0073] Furthermore, the synchronous display of commentary uses the WebVTT subtitle format and is bound to the panoramic player timeline to achieve dynamic text matching. The corresponding commentary is automatically displayed as the panoramic view angle changes, which is used to enhance the timeliness of scene information transmission.

[0074] In this embodiment, it should also be noted that the scenic spot details module is also used to provide graphic information cards, audio players and video embed windows;

[0075] Furthermore, graphic information cards use HTML / CSS structured layout and AJAX to dynamically load Markdown or rich text data, displaying the historical background and cultural value descriptions of scenic spots, providing in-depth cultural interpretation.

[0076] Furthermore, the audio player is based on the HTML5 Audio tag or the Howler.js audio library, supports MP3 / WAV format streaming, and plays professionally recorded scenic spot commentary to assist users in auditory immersion.

[0077] Furthermore, the video embed window integrates Video.js or H5 native Video components, supports HLS / MPEG-DASH adaptive streaming, and can embed attraction promotional videos or documentaries to dynamically display the real scenery and activity scenes of the attractions.

[0078] In this embodiment, it should also be noted that the navigation interaction module is also used to provide travel route recommendations, a search box, and a VR mode switching button;

[0079] Furthermore, travel route recommendations generate a recommendation list based on tag classification and collaborative filtering algorithms, displaying customized travel routes by theme to meet users' personalized travel needs;

[0080] Furthermore, the search box uses the Elasticsearch full-text search engine to implement keyword fuzzy matching and word segmentation queries, allowing users to enter attraction names or keywords to quickly locate target content for information retrieval;

[0081] Furthermore, the VR mode switch button calls the WebXR API or Cardboard SDK to switch to full-screen stereo rendering mode, enabling a VR helmet-compatible display to provide a fully immersive scene browsing experience.

[0082] In this embodiment, it should also be noted that the information acquisition and processing module further includes a panoramic image acquisition unit and a multi-source data cleaning unit;

[0083] Furthermore, the panoramic image acquisition unit is used for panoramic image acquisition through drones and fixed-point photography;

[0084] Furthermore, the multi-source data cleaning unit is used to unify the format and remove redundancy of multi-source data;

[0085] In this embodiment, it should also be noted that the panoramic stitching and fusion module also includes an image alignment unit, a seamless stitching unit, and a multimodal fusion engine;

[0086] Furthermore, the image alignment unit extracts key feature points from images of different viewpoints through SIFT, ORB, or SURF feature point matching and calculates the geometric transformation matrix to align panoramic images taken from multiple angles, eliminating perspective differences and providing a geometric consistency basis for seamless stitching.

[0087] Furthermore, the seamless stitching unit uses the image stitching algorithm of OpenCV or PanoTools library to smoothly stitch the aligned images together through multi-band fusion or exposure compensation, eliminating color differences and distortion at the seams and generating a complete panoramic image.

[0088] Furthermore, the multimodal fusion engine uses natural language processing and audio timestamp synchronization technology to bind text commentary, background music and panoramic images, dynamically linking text / audio information with the panoramic scene, enhancing the immersive experience. Clicking on hotspots triggers voice explanations.

[0089] In this embodiment, it should also be noted that the map data management module also includes a map drawing engine, a database and a spatial index unit;

[0090] Furthermore, the map drawing engine uses the coordinate mapping algorithm of ArcGIS or QGIS to map the latitude and longitude coordinates of scenic spots to the pixel positions of the painted map, ensuring the consistency between the painted map and the real geographic space, and supporting accurate labeling and navigation of hotspots.

[0091] Furthermore, the database uses MySQL to store structured metadata such as attraction names, coordinates, and descriptions, while MongoDB stores unstructured data such as panoramic image paths and audio files, enabling efficient management of multi-source heterogeneous data and supporting fast query and expansion.

[0092] Furthermore, the spatial index unit builds a spatial index based on the R-tree or GeoHash algorithm, and combines it with the quadtree to optimize geographic range queries, which is used to quickly locate hotspots around the user or in a specified area, improving navigation response speed;

[0093] In this embodiment, it should also be noted that the service interface module and the operation and maintenance deployment module also include RESTfulAPI, WebSocket and WeChat applet SDK integration units;

[0094] Furthermore, the RESTful API is based on the SpringBoot or Flask framework to design a standardized interface, using JSON format to transmit data, and is used to provide core services such as scenic spot information query, panorama loading, and user behavior recording;

[0095] Furthermore, WebSocket establishes a two-way communication channel through Socket.IO or the native WebSocket protocol, which is used to push navigation path updates and user location synchronization dynamic information in real time;

[0096] Furthermore, the WeChat Mini Program SDK integration unit calls WeChat's official API to integrate login authorization, map components, and payment functions, enabling seamless integration between the Mini Program and backend services, supporting one-click sharing and social dissemination.

[0097] In this embodiment, it should also be noted that the operation and maintenance deployment module also includes a load balancing unit, a cloud server, and a log monitoring unit;

[0098] Furthermore, the load balancing unit uses the Nginx reverse proxy configuration to configure polling or weighted least connection algorithms to distribute user requests to multiple servers to avoid single points of failure and improve system stability and throughput in high-concurrency scenarios.

[0099] Furthermore, the cloud server deploys backend services based on Alibaba Cloud ECS or Tencent Cloud CVM, and combines object storage to host large panoramic files, providing elastic computing resources and high-availability storage, supporting dynamic expansion and contraction.

[0100] Furthermore, the log monitoring unit collects log data through Elasticsearch and combines it with Kibana to visualize the response time and error rate indicators of the monitoring interface, which is used to diagnose system anomalies in real time, optimize performance bottlenecks, and ensure service reliability.

[0101] Example 2, please refer to Figure 2 In practical applications, the implementation method of the front-end framework of the panoramic image multi-source information fusion system specifically includes the following steps:

[0102] S11. User Interaction and Interface Display: The painted map interface uses GIS geocoding technology to delineate administrative boundaries and render interactive hotspots via Canvas or WebGL. In this embodiment, the GIS geocoding technology can be the Tencent Maps API. When a user enters the WeChat mini-program, the painted map interface loads. The regional information bar dynamically displays the distribution of cultural and tourism resources in each administrative district. When a user clicks a flashing hotspot or selects a location, the frontend requests the corresponding attraction data from the backend via a RESTful API.

[0103] S12. Panoramic display: Use the WebGL rendering engine of the Three.js or Marzipano library to parse panoramic images. The front-end receives the panoramic image URL and associated data returned by the back-end. The panoramic player loads and renders the 360° image, supporting user dragging and zooming. The scene switching button triggers the perspective jump through the front-end routing, for example: from the main facade to the aerial view.

[0104] S13. Interactive display of attraction details: AJAX dynamically loads Markdown text and multimedia resources, and HTML5 Audio / Video components process audio and video. When a user clicks on an attraction name, the front-end requests attraction details. Graphical information cards display historical background, the audio player loads commentary, and a promotional video is embedded in the video window.

[0105] S14. Navigation interaction: Elasticsearch implements keyword segmentation retrieval, and the collaborative filtering algorithm generates a recommendation list. The user enters keywords, and the front-end calls the Elasticsearch interface to return matching results. Based on the user's historical behavior data, the collaborative filtering algorithm recommends personalized travel routes.

[0106] Example 3: In this embodiment, the implementation method of the backend framework of the panoramic image multi-source information fusion system specifically includes the following steps:

[0107] S21. Information Collection and Processing:

[0108] Panoramic image acquisition: drones can take aerial photos to generate panoramic views from a high altitude. Fixed-point equipment can capture multi-angle images. Drones can collect panoramic images of scenic spots along preset paths. Fixed-point shooting equipment can capture photos from multiple perspectives, including the front elevation and side elevation.

[0109] Multi-source data cleaning, FFmpeg conversion of audio and video formats, OpenCV unified image resolution, original data image size normalized to 4096×2048, and audio sampling rate unified to 44.1kHz;

[0110] Redundant data filtering to remove blurry images and invalid audio clips;

[0111] S22. Panoramic stitching and fusion:

[0112] Scale space extreme value detection, see formula (1):

[0113] L(x,y,σ)=G(x,y,σ)*I(x,y)(1);

[0114] Where G(x,y,σ) is the Gaussian kernel function used to smooth the image at different scales (σ), which is expressed as formula (2):

[0115]

[0116] Where I(x,y) is the pixel value matrix of the original image, * represents the convolution operation, which eliminates noise and extracts multi-scale features through Gaussian kernel filtering, and σ is the scale parameter that controls the degree of smoothness.

[0117] Key point positioning and screening, see formula (3):

[0118]

[0119] Where H is the Hessian matrix, which represents the second-order derivative of the local area of the image and is used to describe the curvature characteristics of the key points, as shown in formula (4):

[0120]

[0121] Where Tr(H) is the trace of the matrix, reflecting the sum of the curvatures; Det(H) is the determinant of the matrix, reflecting the product of the curvatures; r is the threshold ratio, which is 10 in this embodiment and is used to exclude edge response points, fences, and branches that are prone to mismatching.

[0122] Seamless splicing and multi-band fusion, see formula (5):

[0123]

[0124] Where, I k (x, y) is the image data of the kth frequency band (low frequency, medium frequency, high frequency);

[0125] w k The weight of each frequency band is usually adjusted dynamically according to the amount of information in the frequency band. For example:

[0126] Low frequency (overall brightness): weight w1 = 0.5, retaining smooth transition;

[0127] Mid-frequency (edge structure): weight w2 = 0.3, enhancing outline clarity;

[0128] High frequency (detailed texture): weight w3 = 0.2 to reduce the influence of noise;

[0129] In actual optimization, the weights are dynamically allocated by calculating the image entropy value, as shown in formula (6):

[0130]

[0131] In the formula, Entropy(I k )=-∑p i log p i ,p i is the pixel grayscale distribution probability;

[0132] Multimodal fusion engine, NLP parsing commentary, timestamp synchronization of audio and scene, word segmentation of text commentary and alignment with the panoramic timeline, and dynamic adjustment of background music volume according to scene switching nodes;

[0133] S23. Spatial index, GeoHash encoding converts geographic coordinates into strings, supporting fast range queries;

[0134] In practical applications, the GeoHash of a scenic spot in Wabu City (32.9356°N, 117.3535°E) is taken as "wtw3sv" as an example, as a specific embodiment of the present invention, GIS coordinate mapping is shown in formula (7):

[0135] x=R·(λ-λ0)·cos(φ0), y=R·(φ-φ0) (7);

[0136] Where φ and λ are the latitude and longitude of the scenic spot, φ0 and λ0 are the latitude and longitude of the center point of the painted map, and R is the radius of the earth (assuming R = 6371 km). In actual application, the scale is adjusted according to the map zoom level. In this embodiment, the center point is (32.9356°N, 117.3535°E), and the coordinates of a scenic spot are (32.9400°N, 117.3600°E). Then:

[0137] x=6371·(117.3600-117.3535)·cos(32.9356)≈0.5km

[0138] y=6371·(32.9400-32.9356)≈0.3km

[0139] The final conversion to pixel coordinates depends on the map resolution. Specifically, one embodiment is: 1 pixel = 10 meters;

[0140] S24. Service interface and operation and maintenance deployment:

[0141] RESTful API, SpringBoot framework design standardized interface, response time ≤ 50ms;

[0142] Example API: GET / api / scene / {id} returns a panoramic view of a scenic spot and its metadata.

[0143] Load balancing weight calculation, see formula (8):

[0144]

[0145] Where, CPU idle rate i is the CPU idle percentage of the i-th server. In practical applications, 80% idle is represented by 0.8;

[0146] Memory idle rate i is the percentage of free memory of the i-th server;

[0147] Dynamically adjust the weights every 5 seconds to prioritize assigning requests to servers with more sufficient resources. The A* path planning is shown in formula (9):

[0148] f(n)=g(n)+h(n) (9);

[0149] Where g(n) is the actual cost of moving from the starting point to node n;

[0150] h(n) is a heuristic function that estimates the optimal cost from node n to the end point. Specifically, see formula (10):

[0151]

[0152] In the actual optimization of the cultural tourism map, g(n) is combined with the road congestion coefficient α, as shown in formula (11):

[0153] g(n) = ∑ road segment length·(1+α·congestion level) (11);

[0154] In this embodiment, it should also be noted that in step S22, the fusion and synchronization of the multimodal fusion engine, in actual application, the audio timestamps are synchronized, and the time error constraint is shown in formula (12):

[0155] t audio -t 全景 |<100 ms (12);

[0156] t audio is the audio playback timestamp, tpanorama is the panoramic player progress timestamp, which is calibrated in real time through the currentTime attribute of the WebAudioAPI;

[0157] Text-scene dynamic binding, keyword extraction, (13):

[0158] Keyword weight = TF-IDF(w)· Position weight(w) (13);

[0159] TF-IDF(w) is term frequency minus inverse document frequency, which measures the importance of a word. Position weight (w) is based on the position of the word in the description.

[0160] Example 4, please refer to Figure 4 In this embodiment, the panoramic image multi-source information fusion method based on the front-end and back-end framework specifically includes the following steps:

[0161] (1) Data collection and preprocessing:

[0162] (1.1) Panoramic image acquisition: UAV aerial photography generates high-altitude bird's-eye view panoramic images (resolution 8K), and fixed-point equipment captures multi-view images (6 orientations × 3 observation points per point);

[0163] (1.2) Multi-source data cleaning: The image format was unified into JPEG2000, the audio was converted into MP3 (bitrate 128kbps), and blurred images were removed using OpenCV (Laplacian variance < 100);

[0164] (2) Panoramic stitching and multimodal definition:

[0165] (2.1) Feature point matching:

[0166] SIFT extracts 500 to 1000 key points from each image, and the RANSAC algorithm removes mismatched points and calculates the homography matrix H:

[0167] Where x i and x′ i is a matching point pair;

[0168] (2.2) Multi-band fusion: The image is decomposed into three frequency bands (low frequency, medium frequency, and high frequency). The low frequency parts are directly spliced together, and the medium and high frequency parts are weighted fused (weight 0.7→0.3);

[0169] (2.3) Commentary and audio quality: BERT model is used to extract commentary keywords, segmented by timeline, and audio timestamps are synchronized with the progress of the panoramic player with an error of <100ms.

[0170] (3) Map data integration and storage:

[0171] (3.1) GIS coordinate mapping: Map the latitude and longitude of scenic spots to the pixel coordinates of the painted map, with an error of ≤5 pixels;

[0172] (3.2) Database storage: It should be noted that the MySQL table structure is:

[0173]

[0174]

[0175] MongoDB stores unstructured data such as panoramic image paths and audio files;

[0176] (4) Front-end and back-end interaction and user request processing:

[0177] (4.1) User request process:

[0178] The front-end initiates a [GET / api / scenes? keyword=XX] request;

[0179] The backend retrieves matching attractions through Elasticsearch and returns a JSON response:

[0180]

[0181] (4.2) Real-time navigation updates: WebSocket pushes the user's location to the backend, updates the navigation path, and calculates the optimal route through the path planning algorithm;

[0182] (5) Immersive experience optimization:

[0183] (5.1) VR mode rendering: Call the WebXR API to render binocular stereo views, with a frame rate of ≥90fps and a field of view (FOV) of 110°, compatible with mainstream VR devices;

[0184] (5.2) Performance monitoring and tuning: Elasticsearch log analysis interface response time distribution;

[0185] (5.3) Dynamic expansion of load balancing: When CPU utilization is greater than 80%, new cloud server instances are automatically added.

[0186] Throughout this specification, references to the terms "one embodiment," "example," "specific example," and the like indicate that the specific features, structures, materials, or characteristics described in conjunction with that embodiment or example are included in at least one embodiment or example of the present invention. In this specification, schematic representations of these terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0187] The preferred embodiments of the present invention disclosed above are intended only to help illustrate the present invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the present invention to the specific embodiments described. Obviously, many modifications and variations are possible based on the content of this specification. These embodiments are selected and described in detail in this specification to better explain the principles and practical applications of the present invention, thereby enabling those skilled in the art to better understand and utilize the present invention. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. Panoramic image multi-source information fusion system, characterized by: include: A front-end framework, which runs on the WeChat applet and includes a painted map module, a panoramic display module, a scenic spot details module, and a navigation interaction module; The painted map module is used to macroscopically display the distribution of cultural and tourism resources in the entire region and provide a navigation portal; The panoramic display module is used to display panoramic images of a single scenic spot at the meso-level, supporting 360° interactive browsing; The attraction details module is used to display text, pictures, audio, and video multi-source information of specific attractions at a micro level; The navigation interaction module is used to provide users with path planning, search and immersive experience functions; It is characterized by further comprising: Backend framework, which includes information collection and processing module, panoramic stitching and fusion module, map data management module, service interface module and operation and maintenance deployment module; The information acquisition and processing module is used to integrate panoramic images, text, audio, and video multi-source data to complete standardization processing; The panoramic stitching and fusion module is used to seamlessly stitch panoramic pictures taken from multiple angles and fuse other media information; The map data management module is used to manage the painted map data, scenic spot coordinates and related information; The service interface module is used to provide an API interface called by the front end to support real-time response of data interaction; The operation and maintenance deployment module is used to ensure stable operation of the system and support high concurrent access.

2. The panoramic image multi-source information fusion system according to claim 1, characterized in that: The painted map module is also used to provide an area information bar, a point selection bar and a flashing hotspot; The regional information bar divides administrative division boundaries based on GIS geocoding technology and displays the distribution of cultural and tourism resources in each administrative district, which is used by users to locate the target area; The point selection bar uses the Vue / React framework to dynamically render the list, combines JSON data to bind the tour route sequence, and arranges the entrances of scenic spots according to the preset optimal tour route to guide users to browse the global scenic spots in order; The flashing hotspot uses Canvas or WebGL to draw interactive hot zones, and realizes dynamic annotation through JavaScript event monitoring, marking the name of the scenic spot in the high-altitude bird's-eye view panoramic map, so that the user can click to jump to the corresponding scenic spot panoramic interface.

3. The panoramic image multi-source information fusion system according to claim 2, characterized in that: The panoramic display module is also used to provide a panoramic image player, scene switching buttons and commentary synchronous display; The panoramic image player is based on the Three.js or Marzipano panoramic rendering library, supports WebGL hardware acceleration, loads and renders 360° panoramic images, and allows users to realize scenes by dragging, dropping, and zooming. The scene switching button is linked to the panoramic SDK interface through the front-end routing to trigger the perspective switch and jump between different observation points. The observation points include the front elevation and the high-altitude view, which are used to display the full view of the scenic spot from multiple angles. The synchronous display of the commentary adopts the WebVTT subtitle format and is bound to the timeline of the panoramic player to achieve dynamic text matching. The corresponding commentary is automatically displayed as the panoramic viewing angle changes, which is used to enhance the timeliness of scene information transmission.

4. The panoramic image multi-source information fusion system according to claim 3, characterized in that: The attraction details module is also used to provide graphic information cards, audio players and video embed windows; The graphic information card uses HTML / CSS structured layout and combines AJAX to dynamically load Markdown or rich text data to display the historical background and cultural value of the scenic spot, providing cultural interpretation; The audio player is based on the HTML5 Audio tag or the Howler.js audio library, supports MP3 / WAV format streaming media playback, and plays professionally recorded scenic spot commentary dubbing to assist users in auditory immersion experience; The video embedding window integrates Video.js or H5 native Video components, supports HLS / MPEG-DASH adaptive streaming, and embeds scenic spot promotional videos or documentaries to dynamically display the real scenery and activity scenes of the scenic spot.

5. The panoramic image multi-source information fusion system according to claim 4, characterized in that: The navigation interaction module is also used to provide travel route recommendations, a search box, and a VR mode switching button; The travel route recommendation generates a recommendation list based on tag classification and collaborative filtering algorithm, and displays customized travel routes by theme classification to meet the user's personalized tour needs; The search box uses the Elasticsearch full-text search engine to implement keyword fuzzy matching and word segmentation queries, allowing users to enter scenic spot names or keywords to quickly locate target content for information retrieval; The VR mode switch button calls WebXRAPI or CardboardSDK to switch to full-screen stereo rendering mode, enabling virtual reality helmet-compatible display to provide fully immersive scene browsing.

6. The panoramic image multi-source information fusion system according to claim 5, characterized in that: The information acquisition and processing module also includes a panoramic image acquisition unit and a multi-source data cleaning unit; The panoramic image acquisition unit is used to acquire panoramic images through drones and fixed-point shooting; The multi-source data cleaning unit is used to unify the format of multi-source data and remove redundancy.

7. The panoramic image multi-source information fusion system according to claim 6, characterized in that: The panoramic stitching and fusion module also includes an image alignment unit, a seamless stitching unit and a multimodal fusion engine; The image alignment unit extracts key feature points of images from different perspectives by matching SIFT, ORB or SURF feature points, and calculates a geometric transformation matrix for aligning panoramic pictures taken from multiple angles; The seamless stitching unit uses an image stitching algorithm based on OpenCV or PanoTools library to smoothly stitch the aligned images through multi-band fusion or exposure compensation, eliminating color differences and distortion at the seams to generate a complete point-to-point panoramic image. The multimodal fusion engine uses natural language processing and audio timestamp synchronization technology to bind text commentary, background music and panoramic images, so as to achieve dynamic association between text / audio information and panoramic scenes, and support clicking hotspots to trigger voice explanations.

8. The panoramic image multi-source information fusion system according to claim 7, characterized in that: The map data management module also includes a map drawing engine, a database and a spatial index unit; The mapping engine uses ArcGIS or QGIS’s coordinate mapping algorithm to map the latitude and longitude coordinates of scenic spots to the pixel positions of the painted map, ensuring the consistency between the painted map and the real geographic space and supporting accurate labeling and navigation of hotspots. The database uses MySQL to store the name, coordinates, and description structured metadata of scenic spots, and MongoDB to store panoramic image paths and unstructured audio file data, which is used to manage multi-source heterogeneous data and support fast query and expansion; The spatial index unit constructs a spatial index based on the R-tree or GeoHash algorithm, and optimizes geographic range queries in combination with the quadtree, so as to locate hot spots around the user or in a specified area.

9. The panoramic image multi-source information fusion system according to claim 8, characterized in that: The service interface module and operation and maintenance deployment module also include RESTfulAPI, WebSocket and WeChat applet SDK integration units; The RESTful API is based on a standardized interface designed with SpringBoot or Flask frameworks, uses JSON format to transmit data, and is used to provide core services such as scenic spot information query, panorama loading, and user behavior recording. The WebSocket establishes a two-way communication channel through one of the Socket.IO or native WebSocket protocols for real-time push navigation path updates and user location synchronization dynamic information; The WeChat mini program SDK integration unit calls the WeChat official API to integrate login authorization, map components, and payment functions to achieve seamless docking between the mini program end and the backend service and support one-click sharing and social dissemination.

10. The panoramic image multi-source information fusion system according to claim 9, characterized in that: The operation and maintenance deployment module also includes a load balancing unit, a cloud server and a log monitoring unit; The load balancing unit configures polling or weighted least connection algorithm through Nginx reverse proxy to distribute user requests to multiple servers to avoid single point failure; The cloud server deploys backend services based on Alibaba Cloud ECS or Tencent Cloud CVM, and combines object storage to host large panoramic files, providing elastic computing resources and high-availability storage, and supporting dynamic expansion and contraction; The log monitoring unit collects log data through Elasticsearch and combines it with Kibana to visualize the response time and error rate indicators of the monitoring interface for real-time diagnosis of system anomalies.

Images