Multi-modal spatio-temporal data format GSS rendering method, rendering engine, equipment and medium

Abstract

The invention discloses a multi-mode spatio-temporal data format GSS rendering method, a rendering engine, equipment and a medium. The method comprises the following steps: constructing story information, story animation and story element data; storing the story information, the story animation and the story element data as a multi-mode spatio-temporal data format GSS; on the basis of story logic, calling GSS data, and using a rendering engine to render and express a story; when the stories are expressed, base map elements in the story elements are visually presented in a map mode. According to the method, efficient storage, rapid retrieval and ordered management of story-related multi-modal data are realized by designing a multi-modal spatio-temporal data format specially adapted to story logic; according to the technical scheme, a rendering engine matched with the multi-modal spatial-temporal data is researched and developed, the multi-modal spatial-temporal data is accurately analyzed and rendered on the basis of a map, and dynamic visual presentation and interactive operation of story position information are realized, so that the expression mode of a story is innovated, and the immersion and participation sense of a user in the story reading and watching process are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of story expression based on geographic information and multi-modal data storage, and in particular to a multi-modal spatio-temporal data format GSS rendering method, a rendering engine, a device and a medium. BACKGROUND

[0002] With the rapid development of information technology, the fusion application of geographic information system (GIS) and multi-modal data has gradually become a research hotspot. In traditional story telling and information display, the presentation method of location elements is relatively single, mainly relying on static picture insertion or link jump. However, when a story or information contains a large amount of location information and there are frequent changes in location, this presentation method has many defects. For example, static pictures can only show the scene in a single time and space dimension, and cannot dynamically and continuously display the spatio-temporal evolution process of the location in the story, lack of interactivity, and it is difficult for readers or audiences to intuitively perceive the dynamic changes of the story scene; and link jump will break the coherence of the story narration, and destroy the immersion and reading experience of the user.

[0003] In recent years, multi-modal story expression systems have gradually emerged, which present time and location elements in a map visualization manner, supplemented by text, pictures, charts, videos and other multimedia elements, making the story more vivid and three-dimensional, and improving the readability and appeal of the story.

[0004] For example, Chinese patent document CN109542389B discloses an audio effect control method for multi-modal story content output, which can change the output audio effect according to different story content and story scenes, cooperate with the development of the plot, and improve the user's experience of obtaining content such as picture book reading. However, this method mainly focuses on the simple display of information, lacks a unified storage and management standard for multi-modal spatio-temporal data, and cannot meet the needs of ordered organization and visual expression of complex story logic.

[0005] In addition, Chinese patent document CN118917407A proposes a large language model driven multi-modal opinion expression recognition system and method, which realizes an intelligent recognition solution for multi-modal opinion expression in real-world scenarios. However, this method mainly focuses on data storage and retrieval, and does not involve dynamic visual presentation and interactive operation of data. In the rendering technology of geographic information and multi-modal data fusion, existing technologies mostly focus on the display of a single data type or simple map annotation, lack comprehensive processing and rendering capabilities for multi-modal spatio-temporal data, and cannot realize dynamic visual presentation and interactive operation of story location information.

[0006] Therefore, the prior art has deficiencies in storage, management and rendering of multi-modal spatio-temporal data, and cannot meet the needs of efficient integration and dynamic display of multi-modal spatio-temporal data in the fields of education and teaching, digital publishing, film and television creation, and tourism promotion. In reality, a story expression scheme based on a multi-modal spatio-temporal data format and a rendering engine is needed. By designing a multi-modal spatio-temporal data format specially adapted to the story logic, efficient storage, rapid retrieval and orderly management of multi-modal data related to the story are realized. A rendering engine is developed to accurately analyze and render the multi-modal spatio-temporal data based on a map, realize dynamic visual presentation and interactive operation of the story location information, thereby innovating the expression of the story and improving the immersion and participation of users in the process of reading and watching the story. SUMMARY

[0007] In view of the above problems, the purpose of the present application is to provide a multi-modal spatio-temporal data format GSS rendering method, a rendering engine, a device and a medium, which accurately analyze and render multi-modal spatio-temporal data based on a map, realize dynamic visual presentation and interactive operation of story location information.

[0008] The present application provides a multi-modal spatio-temporal data format GSS rendering method, a rendering engine, a device and a medium.

[0009] The first aspect is a multi-modal spatio-temporal data format GSS rendering method, comprising:

[0010] S1, constructing story information, story animation and story element data;

[0011] S2, saving the story information, story animation and story element data as a multi-modal spatio-temporal data format GSS;

[0012] S3, based on the story logic, calling the GSS data and using the rendering engine to render and express the story;

[0013] When expressing the story, the bottom map element in the story element is presented visually on the map.

[0014] Optionally, the multi-modal spatio-temporal data format GSS comprises:

[0015] The story information data is stored in an element field data format;

[0016] The story animation data is stored in an animation field data format;

[0017] The story information data is stored in a story field data format.

[0018] Optionally, the story information data comprises story basic information and story page data, wherein:

[0019] The story basic information data is stored in the form of an object;

[0020] The story page data is stored in the form of an array;

[0021] The animation data is stored in the form of an array;

[0022] The element data is stored in the element data list of the story page;

[0023] Optionally, the story element data types include multimedia elements, floating window elements, and background elements.

[0024] Second aspect: a multi-modal spatio-temporal data format GSS rendering engine, comprising:

[0025] A data receiving layer for obtaining GSS data from a server;

[0026] A data parsing layer for parsing the GSS data and converting the data into a data format recognizable and processable by the rendering engine;

[0027] A rendering core layer for performing map rendering, element rendering, and animation rendering according to the recognizable and processable data format;

[0028] A user interaction layer for accepting user interaction operations, feeding back the user operation results to the rendering core layer, updating the page display, and enhancing the user participation.

[0029] Optionally, the rendering core layer performs map rendering, and the steps include:

[0030] Three-dimensional map rendering is based on Cesium.js to create a three-dimensional earth scene, and two-dimensional map rendering is based on Leaflet to load vector and raster data of the earth / map basic data layer, initialize the map according to the background element configuration information, and update the map view and display content in real time when the story page is switched or the element position is changed.

[0031] The rendering method according to claim 1, wherein the rendering engine:

[0032] The front-end framework selects Vue.js as the core front-end framework;

[0033] The map rendering library integrates Cesium.js / Leaflet as the map rendering engine;

[0034] The animation library uses GSAP to realize animation effects;

[0035] Multimedia processing, with the help of HTML5 tags, combines API to display and play multimedia data.

[0036] Optionally, the rendering engine employs a data caching technology to cache the acquired data, sets a cache expiration policy, and reduces the number of data requests and loading time.

[0037] Third aspect: An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of the method provided in the first aspect when executing the program.

[0038] Fourth aspect: A non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program is executable by a processor to implement the steps of the method provided in the first aspect.

[0039] Advantages of the present application:

[0040] 1. The present application provides a story expression scheme based on a multi-modal spatio-temporal data format and a rendering engine. By designing a multi-modal spatio-temporal data format specially adapted to story logic, efficient storage, fast retrieval, and orderly management of story-related multi-modal data are achieved. A rendering engine is developed to support this, which is based on a map for accurate analysis and rendering of multi-modal spatio-temporal data, achieving dynamic visual presentation and interactive operation of story location information, thereby innovating the expression of stories and enhancing the immersion and participation of users during story reading and viewing.

[0041] 2. The GSS data format of the present application provides a unified storage and management standard for multi-modal spatio-temporal data, breaking down barriers between different types of data, and enabling efficient integration of geographic information, multimedia resources, story text, and other multi-modal data, avoiding the management confusion and data redundancy problems caused by scattered data storage. The systematic division and storage of story, animation, and element data make the data structure clear, easy to manage and extend.

[0042] 3. The present application can achieve fast retrieval and accurate positioning of multi-modal spatio-temporal data through reasonable data structure design and indexing mechanism. In processing large-scale data, it can significantly improve the efficiency of data query and acquisition, providing strong data support for story creation and display. It enables the system to quickly locate to a specific page or animation, improving user experience.

[0043] 4. The rendering engine of the present application is based on the Earth / map, organically combines the location information in the story with multimedia elements, and realizes the dynamic visual presentation of the story scene. Users can intuitively feel the changes in geographical location and scene switching in the story, enhancing the immersion and appeal of the story. Through the integration of map visualization and multimedia elements, users can clearly see the marching route of the caravan and the location of important events, as if they were there, enhancing the immersion of the story.

[0044] 5、The application supports multiple interaction modes, and users can perform personalized operations according to their own needs, such as adjusting the animation playing speed, switching the story perspective, and the like. This interactive story expression mode breaks the linear mode of traditional story telling, gives users more participation rights and selection rights, and improves the user's reading experience. Users can trigger detailed information display by clicking on the locations on the map, or view the story content at different stages by dragging the timeline slider, and innovative story expression modes are provided.

[0045] 6、The application adopts advanced rendering techniques and optimization strategies, such as data caching, resource preloading, animation optimization, and the like, can reduce system resource consumption while ensuring high-quality rendering effect, and improve rendering efficiency. Even when processing complex multi-modal data and large-scale scenes, the smooth operation of the system can be ensured. The Cesium.js map rendering is optimized by block loading and lazy loading strategy, the map rendering precision and details are dynamically adjusted, and the rendering effect and performance consumption are balanced.

[0046] 7、The method of the application has undergone comprehensive compatibility testing and processing, and the rendering engine can stably run on mainstream browsers and various devices, including desktops, mobile terminals, and the like. This makes the story expression system developed based on the rendering engine have a wider audience group, improves the usability and popularity of the system, and optimizes the touch interaction experience. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 is a flowchart of the multi-modal spatio-temporal data format GSS rendering method of the application;

[0048] Figure 2 is a structural diagram of the rendering engine of the application;

[0049] Figure 3 is a structural diagram of the multi-modal spatio-temporal data format GSS of the application;

[0050] Figure 4 is a principle flowchart of the rendering engine of the application;

[0051] Figure 5 is a structural diagram of the electronic device of the application. DETAILED DESCRIPTION

[0052] The embodiments of the application will be described in detail below, examples of which are shown in the drawings, wherein the same or similar symbols represent the same or similar elements or elements with the same or similar functions throughout. The embodiments described below by referring to the drawings are exemplary and are only used to explain the application, and cannot be understood as a limitation of the application.

[0053] In traditional storytelling, location, as one of the six elements, is typically presented as a static image or link. However, this approach has limitations when it comes to showing changes in location. It cannot dynamically display the spatial and temporal evolution of the story scene, lacks interactivity, and affects readers' intuitive experience.

[0054] The multimodal storytelling system emphasizes location information, using map visualization and multimedia elements to make stories more vivid and three-dimensional, improving the ease and readability of storytelling. Earth Storytelling provides map resources and animation features to help users create engaging stories and showcase them through preview and sharing functions.

[0055] Existing technologies have deficiencies in the storage, management, and rendering of multimodal spatiotemporal data, and cannot meet the needs of multiple fields for efficient integration and dynamic display.

[0056] In view of the above problems, the present invention provides a multimodal spatiotemporal data format GSS rendering method. Figure 1 A schematic diagram of a process for rendering a multimodal spatiotemporal data format GSS provided by an embodiment of the present invention, the method comprising:

[0057] S1. Construct story information, story animation and story element data.

[0058] The present invention systematically divides the data involved in the story, constructs a multimodal spatiotemporal data format system with story information, story animation and story elements as the core, standardizes the data life cycle process, and improves the data processing capabilities and user experience of the story expression system.

[0059] S2. Save the story information, story animation and story element data as a multimodal spatiotemporal data format GSS.

[0060] Among them, story information data is stored in the story domain data format; story animation data is stored in the animation domain data format; story element data is stored in the element domain data format, together forming the multimodal spatiotemporal data format GSS.

[0061] The first is story information data, which includes basic story information and story page data.

[0062] The basic story information data is stored in the form of JSON objects. The JSON data encoding example is:

[0063] {

[0064] "storyId": "65796e909349ca05240e9594",

[0065] "name": "Qianshan",

[0066] "desc": null,

[0067] "createTime": "2023-12-20 10:18:15",

[0068] "updateTime": "2023-12-20 10:18:15",

[0069] "status": 3,

[0070] "slideMasterId": "1",

[0071] "thumbnail": "https: / / XXX.XXX.com / earthstory / arGccP5QfkFktrYcrmzSp3QTxrRX8c.png"

[0072] }

[0073] The story basic information data is in the form of a JSON object, including the story unique identifier (storyId), story name (name), story description (desc), creation time (createTime), update time (updateTime), status (status), master ID (slideMasterId), and thumbnail address (thumbnail).

[0074] Story page data is stored in the story information as a JSON array, with each element being a story page object.

[0075] An example of JSON data encoding is: [

[0077] {

[0078] "id": "f2w0o0yzayn28e2je9i93seg",

[0079] "name": "Unnamed page",

[0080] "layoutCode": "1",

[0081] "thumbnail": "https: / / XXX.XXX.com / earthstory / iAsE45Rx6hJGntA5ih5RZzzXzPjwaC4i.png",

[0082] "elementDataList": [

[0083] / / Element data (currently empty)

[0084] ],

[0085] "elementGroups": []

[0086] },

[0087] {

[0088] "id": "o2syzyb48q4jdxda9cjnq23w",

[0089] "name": "Unnamed page",

[0090] "layoutCode": "3",

[0091] "thumbnail": "images / page / default_story_page_3d.png",

[0092] "elementDataList": [

[0093] / / Element data (currently empty)

[0094] ],

[0095] "elementGroups": []

[0096] } ]

[0098] The story page data includes the page's unique identifier (id), page name (name), layout code (layoutCode), thumbnail address (thumbnail), element data list (elementDataList), and element group list (elementGroups). The element data list stores information about various elements within the page, and the element group list can be used to group and manage elements.

[0099] Animation data is stored in story information in the form of an array, and each element is an animation object.

[0100] An example of JSON data encoding is: [

[0102] {

[0103] "animationId": "whg1p17v71w6onatcoidw0jy",

[0104] "animationCode": "storyMultimedia",

[0105] "animationConfig": {},

[0106] "sort": null,

[0107] "groupId": null,

[0108] "type": 2,

[0109] "name": "Unnamed page",

[0110] "animationMaterials": [

[0111] {

[0112] "extValues": {},

[0113] "materialType": "1",

[0114] "materialId": "f2w0o0yzayn28e2je9i93seg"

[0115] } ]

[0117] },

[0118] {

[0119] "animationId": "0rmt9rmhf4h0t16t01snm4yc",

[0120] "animationCode": "storyPageDefault",

[0121] "animationConfig": {},

[0122] "sort": null,

[0123] "groupId": null,

[0124] "type": 2,

[0125] "name": "Unnamed page",

[0126] "animationMaterials": [

[0127] {

[0128] "extValues": {},

[0129] "materialType": "1",

[0130] "materialId": "o2syzyb48q4jdxda9cjnq23w"

[0131] } ]

[0133] } ]

[0135] Among them, the animation object contains animation unique identifier (animationId), animation code (animationCode), animation configuration (animationConfig), sorting (sort), group ID (groupId), type (type), name (name) and animation material list (animationMaterials). The animation material list specifies the page or element associated with the animation.

[0136] Element data is stored in the element data list of the story page, and the element data has general attributes and specific element type data configuration.

[0137] Element general attributes include: element data is stored in the element data list of the story page, and each element object contains unique identifier (id), element code (eleCode), type (type), sorting (sort), name (name), extension value (extValues) and data configuration (dataConfig).

[0138] Data configuration contains different contents according to different element types. Specific element type data configuration includes multimedia element (eleCode: "multimedia") data configuration, floating window element (eleCode: "floatWindow") data configuration and base layer element (eleCode: "baseLayer") data configuration.

[0139] Among them, the multimedia element (eleCode: "multimedia") data configuration, the JSON data coding example is as follows:

[0140] {

[0141] "id": "8hag4lg049fcl45xrtn8qnw4",

[0142] "eleCode": "multimedia", ​

[0143] "type": 1,

[0144] "sort": null,

[0145] "name": "Multimedia Resources",

[0146] "dataConfig": {

[0147] "type": "image",

[0148] "url": "https: / / XXX.XXX.com / earthstory / iAsE45Rx6hJGntA5ih5RZzzXzPjwaC4i.png"

[0149] }

[0150] }

[0151] The multimedia element (eleCode: "multimedia") data configuration includes the multimedia type (such as image, audio, video) and resource address (url).

[0152] The data configuration of the floating window element (eleCode: "floatWindow") and the JSON data encoding example are as follows:

[0153] {

[0154] "id": "q9ghui1jezemxwrpxd2q8o2",

[0155] "eleCode": "floatWindow",

[0156] "type": 3,

[0157] "sort": null,

[0158] "name": "Floating Window",

[0159] "dataConfig": {

[0160] "isLock": false,

[0161] "isFolded": true,

[0162] "images": [],

[0163] "bindView": "f2w0o0yzayn28e2je9i93seg",

[0164] "texts": [" "],

[0165] "width": 320,

[0166] "x": 14,

[0167] "y": 60,

[0168] "theme": "default",

[0169] "height": 620

[0170] }

[0171] }

[0172] The data configuration of the floating window element (eleCode: "floatWindow") includes whether to lock (isLock), whether to fold (isFolded), an image list (images), a bound view (bindView), text content (texts), a width (width), an x coordinate (x), a y coordinate (y), a theme (theme), and a height (height).

[0173] The data configuration of the base map element (eleCode: "baseLayer") is as follows:

[0174] {

[0175] "id": "baseLayerId",

[0176] "eleCode": "baseLayer",

[0177] "type": 4,

[0178] "sort": null,

[0179] "name": "Star Map Base Map",

[0180] "dataConfig": {

[0181] "coverLayerId": "daas_annotation",

[0182] "layerType": "GENERAL",

[0183] "thumb": "images / layer_thumb / basemap.jpg",

[0184] "maximumLevel": 18,

[0185] "type": "xyz",

[0186] "filterColor": "#e6eaf2",

[0187] "url":

[0188] "https: / / tiles{s}.XXX.XXX.com / base / v1 / img / {z} / {x} / {y}?format=webp&tmsIds=w&token=&sign=b505fbc54c7e47db53c69c9173d737889c4e8f41ebb0079cd7feacec00b70dba&clientId=7db4955f-2933-445d-900c-3815f006ad99&secretId=kfinv1QCF4y391a6OObn5eueZgl&expireTime=1702733602",

[0189] "subdomains": ["0", "1", "2"],

[0190] "name": "星图底图",

[0191] "id": "basic_img",

[0192] "minimumLevel": 1,

[0193] "gamma": 1.4,

[0194] "zIndex": 2

[0195] }

[0196] }

[0197] The data configuration of the base layer element (eleCode: "baseLayer") includes cover layer ID (coverLayerId), layer type (layerType), thumbnail (thumb), maximum level (maximumLevel), type (type), filter color (filterColor), address (url), subdomain list (subdomains), name (name), ID (id), minimum level (minimumLevel), gamma value (gamma), z-axis index (zIndex), and the base layer element is visually presented on the map when expressing the story.

[0198] Other element types (such as perspective elements, GeoJSON drawing elements, etc.) are also defined in the data configuration with their respective required attributes in a similar manner.

[0199] S3, based on the story logic, calls the GSS data and uses the rendering engine to render and express the story.

[0200] The rendering engine adopts a layered architecture design, including a data receiving layer, a data parsing layer, a rendering core layer, and a user interaction layer.

[0201] The data receiving layer obtains the multi-modal spatio-temporal data GSS from the server through the fetch method, supporting GET and POST request methods. The obtained data is subjected to preliminary integrity verification to ensure that the data contains story basic information, story page data, animation data, and element data, etc., and to ensure the accuracy and availability of the data, providing a reliable data foundation for subsequent processing.

[0202] The data parsing layer parses the story data, animation data, and element data. In terms of story data parsing, the story name, description, thumbnail, and other basic information are extracted for initial page display, and the layout code, thumbnail, and element data list are obtained by parsing the story page data. During animation data parsing, the animation unique identifier, animation code, animation configuration, and animation material list are extracted, and the animation type is determined according to the animation code, and the animation materials are associated with the corresponding page or element. Element data parsing targets different types of elements (multimedia elements, floating window elements, base layer elements, etc.), extracts their required attributes according to element coding and data configuration, such as the type and resource address of multimedia elements, the position and content of floating window elements, and the layer attributes of base layer elements, and converts the original data into a format recognizable and processable by the engine.

[0203] The rendering core layer performs map rendering, element rendering, and animation rendering based on the recognized and processed data format.

[0204] The rendering core layer includes map rendering, element rendering, and animation rendering functions.

[0205] Among them, the three-dimensional map rendering creates a three-dimensional earth scene based on Cesium.js, the two-dimensional map rendering is based on Leaflet, loads vector and raster data of earth / map basic data layer, initializes the map according to the base map element configuration information, and updates the map view and display content in real time when the story page switches or the element position changes.

[0206] Element rendering traverses the story page element data list, and according to the element type, uses HTML5 tags, custom components or related libraries to draw and display, such as creating corresponding tags and setting styles for multimedia elements, creating components and adding interaction events for floating window elements.

[0207] Animation rendering creates animation sequences according to parsed animation data, sets animation parameters and binds them to corresponding elements, and updates page state and element attributes in real time during playback to achieve animation effects.

[0208] User interaction layer provides basic interaction and custom interaction functions.

[0209] Basic interaction supports mouse and touch operations, and realizes map zooming, panning, rotating, and element clicking, dragging and other operations by listening to Cesium.js and HTML5 related events.

[0210] Custom interaction allows users to set custom animation playback speed, switch story view and other personalized operations through the interface, and feedbacks user operation results to the rendering core layer through creating interaction components and event handling functions, updates page display, and enhances user engagement.

[0211] The technology selection of the rendering engine adopts:

[0212] Front-end framework: Choose Vue.js as the core front-end framework; use its responsive data binding and component development features to encapsulate the rendering engine's function modules into components, realize efficient binding of data and view and reuse of components, improve development efficiency and code maintainability.

[0213] Map rendering library: Integrate Cesium.js / Leaflet as the map rendering engine, which has powerful three-dimensional earth rendering capabilities, can display geographic spatial data with high precision, supports complex map interaction operations, and provides technical support for story scene rendering based on earth / map.

[0214] Animation library: Use GSAP to achieve animation effects, which has rich animation functions and high-performance rendering capabilities, can meet the needs of complex animation scenarios, and ensures the smoothness and diversity of animation playback.

[0215] Multimedia handling: with HTML5 native , <audio> 、 <video>Tags, combined with related APIs, display and play multimedia data, enabling seamless integration and flexible operation of multimedia content on Web pages.

[0216] Further, the rendering engine uses data caching technology for performance optimization, caches the obtained data, sets a reasonable cache expiration policy, reduces the number of data requests and loading time, implements resource preloading, preloads multimedia resources at the initial stage of page loading, and uses progressive loading to improve resource utilization efficiency, uses animation optimization techniques such as batch processing animations and hardware acceleration to reduce the impact of animations on browser performance, optimizes data loading strategies for Cesium.js map rendering, uses chunk loading and lazy loading, dynamically adjusts map rendering precision and details, and balances rendering effects and performance consumption.

[0217] Further, compatibility testing is conducted on mainstream browsers (Chrome, Firefox, Safari, Edge, etc.), CSS prefixes, polyfill, and other technologies are used to solve browser feature differences, ensuring that the rendering engine runs normally in various browsers; support for multiple device types, dynamically adjust page layout and element size according to device screen size and resolution through responsive design, and optimize touch interaction experience for mobile devices to ensure consistent user experience on different terminals.

[0218] Application example: Building a GSS data architecture for multi-modal spatio-temporal data, taking the creation of a "Land Silk Road" story as an example, building multi-modal spatio-temporal data format GSS data in various fields.

[0219] Story information data includes story basic information and story page data.

[0220] Among them, the story basic information stores the overall information of the story by creating a JSON object. The story unique identifier is set to "silkRoadStory001", the story name is "Exploring the Land Silk Road", the description can explain the core content of the historical significance of the Silk Road, trade exchanges, and cultural exchanges, the creation time is recorded as the actual creation time, such as "2024-01-0100:00:00", the update time is the same as the creation time if it is not updated, the status is set to "Published", the master ID is determined according to the actual use of the story layout template, and the thumbnail address selects a representative Silk Road scenery picture link, such as "https: / / XXX.XXX.com / silkroad / thumbnail.jpg".

[0221] Story page data is created by building multiple story page objects to showcase the content of different stages of the Silk Road. For example, the start page has a unique identifier of "startPage001", a page name of "Starting Point of the Silk Road - Chang'an", a layout code determined according to the page design, such as "layout001", and a thumbnail link to a picture of a landmark building in Chang'an, such as "https: / / XXX.XXX.com / silkroad / changan_thumbnail.jpg". In the element data list, a text element is added to introduce the historical status of Chang'an, with a unique identifier of "textElement001", an element code of "text", and data configuration containing the text content "Chang'an, as the starting point of the Silk Road, is an important gateway for ancient China's foreign exchange..."; a map element is added to show the location of Chang'an on the map, with a unique identifier of "mapElement001", an element code of "map", and data configuration containing map type, coordinate range, etc. The element group list can group these related elements for easy management. Subsequent pages are set up according to the route of the Silk Road to show different locations (such as Dunhuang, Samarkand, etc.), each page containing corresponding element data to introduce local features.

[0222] Animation data is created to vividly present the trade and cultural exchange scenes of the Silk Road. For example, an animation describing the departure of a caravan from Chang'an is created, with an animation unique identifier of "caravanDepartureAnimation001" and an animation code of "caravanMove" representing the moving animation of the caravan. In the animation configuration, the duration of the animation is set to "5s", the easing effect is set to "linear", the order is set according to the development of the story, the group ID can be set to "tradeGroup001", the type is "element animation", and the name is "caravan departure from Chang'an". The animation material list is associated with the elements representing the caravan (such as the caravan icon element created in the element field) and the starting page to ensure that the animation is triggered in the corresponding scene. Another animation is created to show the construction process of the Dunhuang Mogao Grottoes, with an animation unique identifier of "mogaoCaveConstructionAnimation001" and an animation code of "siteConstruction". The animation is associated with the story page showing Dunhuang and the element representing the Mogao Grottoes to show the gradual construction of the Mogao Grottoes with the prosperity of the Silk Road.

[0223] Element field data includes multimedia elements, floating window elements, and background elements.

[0224] The multimedia element is created for displaying the picture of the scenery along the Silk Road, such as the picture of the Mogao Grottoes in Dunhuang. The unique identifier is set as "mogaoCaveImage001", the element code is "multimedia", the type is 1 (representing the picture type), the order is determined according to the display order, the name is "appearance of the Mogao Grottoes in Dunhuang", the multimedia type in the data configuration is "image", and the resource address is the picture link "https: / / XXX.XXX.com / silkroad / mogao_cave.jpg".

[0225] The floating window element is created for supplementing the introduction information. For example, for the detailed introduction of the Mogao Grottoes in Dunhuang, the unique identifier of the floating window element is "mogaoCavePopup001", the element code is "floatWindow", the type is 3, the order is set according to the user interaction requirement, and the name is "detailed introduction of the Mogao Grottoes in Dunhuang". The data configuration includes whether to lock (set as "false"), whether to fold (set as "true"), a picture list (if there are related pictures, the link can be added), a bound view (associated with the element of the page showing the story of Dunhuang), text content (text introducing the history and artistic value of the Mogao Grottoes in detail), width (set as "300px"), x coordinate (set as "100px"), y coordinate (set as "100px"), theme (set as "default"), and height (set as "400px").

[0226] The base layer element selects a suitable map base layer element to display the geographical position and route of the Silk Road. The unique identifier of the base layer element is "silkRoadMapBaseLayer001", the element code is "baseLayer", the type is 4, the order is at the bottom layer, and the name is "map base layer of the Silk Road". The data configuration includes the overlay ID (such as "mapOverlay001"), the layer type (set as "GENERAL"), the thumbnail (the link of the map thumbnail), the maximum level (set as "18"), the type (such as "xyz"), the filter color (set as "#e6eaf2"), the address (the map tile address "https: / / tiles {s}.XXX.XXX.com / silkroad / map / {z} / {x} / {y}?format=webp..."), the list of subdomains (such as "['0', '1', '2']"), the name (repeated as "map base layer of the Silk Road"), the ID (such as "mapBaseId001"), the minimum level (set as "1"), the gamma value (set as "1.4"), and the z-axis index (set as "1").

[0227] When the rendering engine is used, the rendering engine performs GSS analysis on the multi-modal data. When the system reads the "Overland Silk Road" story data, the multi-modal data analysis module automatically calls the corresponding sub-module for analysis according to the data format identification. The map data is analyzed, and the vector data of the Silk Road map is converted into a graphical object that can be processed by the rendering engine, accurately presenting the location and shape of geographical features such as cities and roads; the picture that shows the history of the Silk Road is analyzed, and the spatial location information in the picture is extracted, such as the corresponding position of the picture shooting location on the map; the text describing the trade and cultural exchange of the Silk Road is processed by natural language processing, and key information such as time (such as the year the Silk Road was opened, the period of trade prosperity, etc.), place (such as the cities passed through, trade distribution centers, etc.), and person (such as important merchants and envoys, etc.) is extracted; the related audio and video are decoded to obtain timestamp (such as the starting time of audio playback, the time sequence of video frames) and other information, and all analyzed data is standardized to unify the coordinate reference and data format.

[0228] The rendering engine renders the GSS data. Based on the Silk Road map, the story location information after analysis is marked on the map through geographic coordinate mapping algorithm, clearly presenting the marching route of the caravan and the location of important events. Dynamic map rendering technology is used to display the scene changes along the Silk Road, such as vegetation changes with seasons, prosperity and decline of cities, etc. according to the story time advancement. Related multimedia data is superimposed in the form of layers in the map scene, when the user browses to the scene where the caravan stays in Dunhuang, the environmental sound effect of Dunhuang is automatically played, the historical pictures of Dunhuang Mogao Grottoes are displayed, or the related video clips are played, realizing the fusion rendering of multi-modal data in the map space, and vividly restoring the historical scene of the overland Silk Road.

[0229] The rendering engine presents the interactive story. The user operates through the user interaction unit, such as clicking Chang'an on the map to trigger the display of detailed information of the location, including the introduction of the important position of Chang'an in the Silk Road, related historical pictures and audio commentary; dragging the timeline slider to view the development of the Silk Road in different stages and the events experienced; zooming and panning the map to observe the geographical landscape along the Silk Road from different perspectives. The system processes the corresponding information according to the user operation instruction, and displays the results in real time on the user interaction unit, realizing the interactive story presentation, enabling the user to deeply experience the historical story of the overland Silk Road.

[0230] The application provides a unified storage and management standard for multi-modal spatio-temporal data at the data management level, eliminates the boundaries between different types of data, realizes efficient integration of multi-modal data such as geographic information, multimedia resources and story text, and solves the problems of management confusion and data redundancy caused by scattered data storage. Unified standard and efficient integration. Through the reasonable design of data structure and index mechanism, fast retrieval and accurate positioning of multi-modal spatio-temporal data are realized. When dealing with large-scale data, the efficiency of data query and acquisition is significantly improved, providing strong data support for story creation and display, realizing fast retrieval and accurate positioning.

[0231] At the story expression level, the application enhances the immersion of the story: the rendering engine is based on the earth / map, and the position information in the story is organically combined with the multimedia elements to realize the dynamic visual presentation of the story scene. Users can intuitively feel the change of geographical position and the switching of scenes in the story, enhancing the immersion and appeal of the story. Innovative story expression method: support multiple interaction modes, users can perform personalized operations according to their own needs, such as adjusting the animation playback speed, switching the story perspective, etc. This interactive story expression method breaks the linear mode of traditional story telling, gives users more participation and selection rights, and improves the user's reading experience.

[0232] At the system performance level, the application realizes high-performance rendering: advanced rendering techniques and optimization strategies such as data caching, resource preloading, animation optimization, etc. are adopted, which can reduce system resource consumption and improve rendering efficiency while ensuring high-quality rendering effect. Even when dealing with complex multi-modal data and large-scale scenes, the system can still run smoothly. Wide compatibility: after comprehensive compatibility testing and processing, the rendering engine can run stably on mainstream browsers and various devices, including desktop and mobile devices. This makes the story expression system developed based on the rendering engine have a wider audience group, improving the usability and popularity of the system.

[0233] The application also provides an electronic device, Figure 5 The structure diagram of the electronic device provided by the embodiment of the application is shown in Figure 5 As shown in the figure, the electronic device can include a processor, a communications interface, a memory and a communications bus, wherein the processor, the communications interface and the memory complete mutual communication through the communications bus. The processor can call the logical instructions in the memory, for example, to execute the following method: (method steps).

[0234] In addition, the logic instructions in the above-mentioned memory can be realized in the form of a software function unit and sold or used as an independent product, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0235] The embodiments of the present application also provide a non-transitory computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the method provided by the above-mentioned embodiments, for example, including: (method steps).

[0236] The device embodiments described above are only schematic, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., they can be located in one place, or distributed on a plurality of network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments. Those skilled in the art can understand and implement without creative labor.

[0237] From the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software and necessary universal hardware platforms, and of course can also be realized by hardware. Based on such understanding, the above technical solutions essentially or the part that contributes to the prior art can be embodied in the form of a software product. The computer software product can be stored in a computer readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments.

[0238] It should be pointed out finally that the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit the same; and although the present application has been described in detail with reference to the foregoing embodiments, it should be appreciated by those skilled in the art that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features thereof can be replaced equivalently; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.< / video> < / audio>

Claims

1. A multimodal spatiotemporal data format GSS rendering method, characterized in that: include: S1, construct story information, story animation and story element data; S2, saving story information, story animation and story element data into a multimodal spatiotemporal data format GSS; S3, based on the story logic, calls GSS data and uses the rendering engine to render and express the story; Among them, when expressing a story, the base map elements in the story elements are presented in a map visualization.

2. The rendering method according to claim 1, wherein: The multimodal spatiotemporal data format GSS includes: Story information data is stored in the story domain data format; Story animation data is stored in the animation domain data format; Story element data is stored in the element domain data format.

3. The rendering method according to claim 2, wherein: The story information data includes basic story information and story page data, wherein: The basic information data of the story is stored in the form of objects; Story page data is stored in array form; Animation data is stored in array form; Element data is stored in the Element Data list on the Story page.

4. The rendering method according to claim 1, wherein: The story element data types include: multimedia elements, floating window elements and base map elements.

5. The rendering method according to claim 1, wherein: A multimodal spatiotemporal data format GSS rendering engine for a rendering method, comprising: Data receiving layer, used to obtain GSS data from the server; The data parsing layer is used to parse GSS data and convert the data into a data format that can be recognized and processed by the rendering engine; The rendering core layer performs map rendering, element rendering, and animation rendering based on the identified and processed data formats; The user interaction layer accepts user interaction operations, feeds back the user operation results to the rendering core layer, updates the page display, and enhances user participation.

6. The rendering method according to claim 5, characterized in that: The rendering core layer performs map rendering, and the steps include: 3D map rendering uses Cesium.js to create a 3D earth scene. 2D map rendering uses Leaflet to load vector and raster data of the earth / map base data layer, initializes the map according to the basemap element configuration information, and updates the map perspective and display content in real time when the story page switches or the element position changes.

7. The rendering method according to claim 5, characterized in that: The rendering engine: The front-end framework uses Vue.js as the core front-end framework; The map rendering library integrates Cesium.js / Leaflet as the map rendering engine; The animation library uses GSAP to achieve animation effects; Multimedia processing uses HTML5 tags and APIs to display and control the playback of multimedia data.

8. The rendering method according to claim 5, characterized in that: The rendering engine adopts data caching technology to cache the acquired data and set a cache expiration policy to reduce the number of data requests and loading time.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the rendering method according to any one of claims 1 to 8 are implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the rendering method according to any one of claims 1 to 8 are implemented.

Images