Method and apparatus for processing space-time data in a meta universe scene, and electronic device

Abstract

The application discloses a kind of spatiotemporal data processing method, device and electronic equipment in meta-universe scene.The method comprises: obtaining the space data of the meta-universe scene where the first user is;Obtain the space trajectory data of the moving unit in the scene within the preset time period;And store the space trajectory data to constitute the first user in the preset time period Recall scene visual data.Using the embodiment of the application can realize a new social record, meet the demand of recording the social process of user in meta-universe scene as social recall.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, specifically to a method, apparatus, electronic device, computer-readable storage medium, and computer program product for spatiotemporal data processing in a metaverse scenario. Background Technology

[0002] With the development of technology, people's social settings are changing, and the forms of recording social processes and content are also changing accordingly. For example, in real-world settings, people use cameras and other devices to record social interactions in the form of photos and videos, saving them frame by frame over time. With the development of Web 2.0 social products, when people communicate through social tools, these tools record past interactions in the form of communication history lists. These communication history lists record interactions over time and in a conversational format, including text, voice, emojis, and various file formats. Additionally, some social platforms allow people to share and communicate by uploading photos and videos recorded offline.

[0003] Metaverse is an open and shared online platform that integrates information technology, communication technology, virtual reality (VR), augmented reality (AR), or mixed reality (MR) technologies. Based on AR, VR, or MR technologies, the Metaverse system provides a 3D space where users can enter directly or as avatars and interact with other users. The Metaverse system offers a new social space, and people have a need to socialize within it and record their interactions as social memories. Therefore, a corresponding technological solution is urgently needed to meet these needs. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a spatiotemporal data processing method, apparatus, electronic device, computer-readable storage medium, and computer program product in a metaverse scenario, which can at least record social processes taking place in the metaverse.

[0005] According to one aspect of the present invention, an embodiment of the present invention provides a method for spatiotemporal data processing in a metaverse scenario, comprising the following steps:

[0006] Obtain spatial data of the metaverse scene where the first user is located;

[0007] Acquire spatial trajectory data of motion units in the scene within a preset time period; and

[0008] The spatial trajectory data is stored to form visual data of the first user's memory scene during the preset time period.

[0009] According to another aspect of the present invention, embodiments of the present invention also provide a method for spatiotemporal data processing in a metaverse scenario, comprising the following steps:

[0010] The system acquires the user's current location data and viewpoint data, wherein the location data includes spatial coordinates and orientation, and the viewpoint data includes spatial angles.

[0011] Obtain memory scene data within a preset time period;

[0012] Based on the user's current location data and perspective data, each frame of the memory scene data is converted into synchronization frame data that needs to be synchronized to the user, frame by frame, according to the timeline of the memory scene data.

[0013] Based on the user's current position and perspective, a three-dimensional space of the memory scene is constructed using the first frame of synchronized frame data;

[0014] The driving parameters of the motion unit in the memory scene are calculated based on adjacent synchronized frame data, and the driving parameters include motion speed and motion direction; and

[0015] The position of the motion unit in the memory scene is changed based on the driving parameters, so that the position and / or state of the motion unit changes from the position and / or state of the previous synchronization frame to the position and / or state of the next synchronization frame.

[0016] According to another aspect of the present invention, embodiments of the present invention also provide a spatiotemporal data processing apparatus in a metaverse scenario, comprising:

[0017] The scene space data acquisition module is configured to acquire the spatial data of the metaverse scene where the first user is located.

[0018] A spatial trajectory data acquisition module, connected to the scene spatial data acquisition module, is configured to acquire spatial trajectory data of motion units in the metaverse scene within a preset time period; and

[0019] The storage module is configured to store the spatial trajectory data to form visual data of the memory scene of the first user within the preset time period.

[0020] According to another aspect of the present invention, embodiments of the present invention also provide a spatiotemporal data processing apparatus in a metaverse scenario, comprising:

[0021] The location data acquisition module is configured to acquire the user's current location data and viewpoint data. The location data includes spatial coordinates and orientation, and the viewpoint data includes spatial angles.

[0022] The scene data acquisition module is configured to acquire memory scene data within a preset time period;

[0023] The data conversion module, which is connected to the location data acquisition module and the scene data acquisition module, is configured to convert each frame of the recalled scene data into synchronized frame data that needs to be synchronized to the user, based on the user's current location data and viewpoint data, and according to the timeline of the recalled scene data.

[0024] A 3D scene construction module, which is connected to the data conversion module, is configured to construct a 3D space of the memory scene based on the user's current position and perspective, using the first frame of synchronized frame data.

[0025] A driving parameter module, connected to the data conversion module, is configured to calculate driving parameters of the motion units in the memory scene based on adjacent synchronized frame data. These driving parameters include motion speed and motion direction.

[0026] A motion unit driving module, which is connected to the driving parameter module, is configured to change the position of the motion unit in the memory scene based on the driving parameters, so that the motion unit changes from the position and / or state of the previous synchronization frame to the position and / or state of the next synchronization frame.

[0027] According to another aspect of the present invention, an electronic device is also provided, which includes a processor and a memory storing computer program instructions, wherein the processor executes the computer program instructions to implement the spatiotemporal data processing method described above.

[0028] According to another aspect of the present invention, embodiments of the present invention also provide a computer-readable storage medium storing computer program instructions, which, when executed by a processor, implement the spatiotemporal data processing method as described above.

[0029] According to another aspect of the present invention, embodiments of the present invention also provide a computer program product, the computer program product including computer program instructions, which, when executed by a processor, implement the spatiotemporal data processing method as described above.

[0030] The embodiments of the present invention can record the user's scenes in the metaverse, thereby obtaining a new kind of social record that meets the need to record the user's social process as a social memory. Attached Figure Description

[0031] To more clearly illustrate the implementation of the embodiments of the present invention, the accompanying drawings of the embodiments of the present invention will be briefly described below.

[0032] Figure 1 This is a schematic diagram of the metaverse system architecture based on servers and terminal devices according to an embodiment of the present invention.

[0033] Figure 2 This is a flowchart of a spatiotemporal data processing method in a metaverse scenario according to an embodiment of the present invention.

[0034] Figure 3 This is a flowchart of a spatiotemporal data processing method in a metaverse scenario according to another embodiment of the present invention.

[0035] Figure 4 This is a flowchart illustrating the process of collecting spatial position data of moving units in a scene when the current time information T is determined to be equal to the starting time T1, according to an embodiment of the present invention.

[0036] Figure 5 This is a flowchart of a spatiotemporal data processing method according to another embodiment of the present invention.

[0037] Figure 6 This is a flowchart of a spatiotemporal data processing method in a metaverse scenario according to another embodiment of the present invention.

[0038] Figure 7 This is a flowchart of a data processing method according to another embodiment of the present invention.

[0039] Figure 8 This is a schematic diagram of a first spatiotemporal data processing device in a metaverse scenario according to an embodiment of the present invention.

[0040] Figure 9 This is a schematic diagram of a second spatiotemporal data processing device in a metaverse scenario according to an embodiment of the present invention.

[0041] Figure 10 This is a schematic diagram of the structure of an electronic device 60 provided according to an embodiment of the present invention. Detailed Implementation

[0042] The principles and spirit of the present invention will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided to make the principles and spirit of the present invention clearer and more thorough, enabling those skilled in the art to better understand and implement the principles and spirit of the present invention. The exemplary embodiments provided herein are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments described herein without inventive effort are within the scope of protection of the present invention.

[0043] Those skilled in the art will recognize that embodiments of the present invention can be implemented as a system, apparatus, device, method, computer-readable storage medium, or computer program product. Therefore, the present invention can be specifically implemented in at least one of the following forms: entirely hardware, entirely software, or a combination of hardware and software. According to specific embodiments of the present invention, the present invention claims protection for a spatiotemporal data processing method, apparatus, electronic device, computer-readable storage medium, and computer program product in a metaverse scenario.

[0044] In this document, terms such as first, second, and third are used only to distinguish one entity (or operation) from another, and are not intended to require or imply any order or relationship between these entities (or operations).

[0045] Embodiments of the present invention can be applied to servers and terminal devices. Please refer to... Figure 1 This diagram schematically illustrates a metaverse system architecture based on servers and terminal devices. The metaverse system architecture includes a server 10 and several terminal devices 20. In some examples, the terminal devices 20 are client devices; for example, they can be dedicated devices such as head-mounted displays (HMDs), smart gloves, clothing, and other smart wearable electronic devices. In other examples, the terminal devices 20 can be general-purpose AR / VR (Augmented Reality / Virtual Reality) devices, such as mobile phones, laptops, laptops, tablets, AR glasses, in-vehicle devices, navigation devices, gaming devices, etc.

[0046] Taking AR helmets or AR glasses as an example, head-mounted displays, machine vision systems, and mobile computers can be integrated into wearable devices. These devices have a display resembling glasses and are worn on the user's head. They transmit augmented reality information to the display or project it onto the user's eyes, enhancing visual immersion. In some examples, AR devices also include cameras, which can be wide-angle, telephoto, or structured light cameras (also known as point cloud depth cameras, 3D structured light cameras, or depth cameras). Structured light cameras, based on 3D vision technology, can acquire planar and depth information of objects. They project light with specific structural features onto the object using a near-infrared laser, then an infrared camera collects the reflected light, which is processed by a processor chip. The calculation principle involves determining the object's position and depth information based on changes in the light signal caused by the object, thus presenting a 3D image. Typical terminal devices, such as mobile phones, display two-dimensional images and cannot show the depth of different locations within the image. Structured light cameras can capture and acquire 3D image information, obtaining not only color and other information at different locations but also depth information, which can be used for AR ranging. Of course, ordinary terminal devices can also acquire 2D images using optical cameras and combine this with deep learning algorithms to obtain depth information, ultimately presenting 3D images. In some examples, terminal device 20 also includes positioning components, such as GPS systems, gyroscopes, and IMU units. Using technologies like SLAM (Simultaneous Localization and Mapping) and V-SLAM (Visual Simultaneous Localization and Mapping), an environmental map is constructed based on the acquired sensor information and environmental image information. Simultaneously, the terminal device's own position is calculated, determining its location within the current environmental map. Alternatively, visual features of the environmental space can be extracted, and based on an existing digital spatial model of the current scene, the terminal device's position can be determined through visual feature matching.

[0047] In some examples, terminal device 20 is equipped with metaverse client software or an application (APP) that has AR / VR capabilities, and integrates modules with network connectivity, such as a Wireless-Fidelity (Wi-Fi) module, a Bluetooth module, and a 2G / 3G / 4G / 5G communication module, to connect to server 10 via the network. Users can log in to their user accounts through the client software or APP installed on terminal device 20. Server 10, in conjunction with the client software or APP on terminal device 20, enables various application scenarios. Server 10 can be a single server, a server cluster consisting of multiple servers, or a cloud server, etc.

[0048] The embodiments of the present invention relate to terminal devices and / or servers. The principles and spirit of the present invention will be explained in detail below through several exemplary embodiments or representative implementations.

[0049] Example 1

[0050] In this embodiment, the metaverse system provides users with a virtual digital space scaled 1:1 with the real-world scene. Users log into their client accounts and enter a specific metaverse scene using a designated avatar. When multiple users enter the same metaverse scene with their designated avatars, they can interact, such as conversing or engaging in activities together. Users can also interact with the scene, such as moving existing items, adding new items, or changing the state of existing items. When a user wants to save their memories in the current metaverse scene, they can select the time-space memory function provided by the metaverse system. When the user selects the time-space memory function, the time-space data in the current metaverse scene can be recorded.

[0051] Figure 2 This is a flowchart of a spatiotemporal data processing method in a metaverse scenario according to an embodiment of the present invention, the method comprising the following steps:

[0052] Step S11: Obtain spatial data of the metaverse scene where the first user is located.

[0053] Step S12: Obtain the spatial trajectory data of the motion unit in the scene within a preset time period.

[0054] Step S13: Store the spatial trajectory data to form visual data of the memory scene of the first user within the preset time period.

[0055] In this embodiment, the user who selected the time-space memory function option in the metaverse scene is referred to as the first user, and other users are referred to as the second user to distinguish them. In step S11, the spatial data of the metaverse scene where the first user is located is obtained. The spatial data includes, for example, the spatial structure data of the three-dimensional space of the metaverse scene and the virtual digital content data in the scene, such as the three-dimensional coordinates, materials, colors, etc. of three-dimensional buildings and three-dimensional object models.

[0056] The time-space recall function option of the metaverse system can optionally include a time parameter. When the first user selects the time-space recall function option, they can set a time period, such as setting a start and end time, or setting a start time and duration. Additionally, the metaverse system includes a reference clock. When recording data according to the user-set time period, the metaverse system uses the time information of the reference clock as the standard. In one embodiment, the reference clock is synchronized with the system time. When the first user determines a time period when selecting the time-space recall function option, the preset time period in step S12 is the user-set time period. If the first user does not set a time period, a system default time period is used as the preset time period in step S12, such as the system defaulting to recording the first n minutes of time-space data in the user's scenario.

[0057] In step S12, the motion unit refers to an object or user character whose position or state can change in the metaverse scene. Examples include objects that can be moved by the user character or whose position or state changes for some reason, such as tables, chairs, and stationery on tables. When acquiring the spatial trajectory data of motion units in the scene, within a preset time period, spatial position data of motion units, including the first user, is collected according to preset conditions. A timestamp is added to each collected data based on reference clock information. Multiple collected spatial position data are serialized in chronological order to form spatial trajectory data with a time axis. When the preset condition is a preset frame rate, the spatial position data of the motion unit is collected according to the preset frame rate, allowing the collection of the spatial position data of each motion unit in each frame. When the preset condition is a position change, it is necessary to monitor whether the position of the motion unit in the scene changes to determine if the condition is met. When the position of the motion unit changes, the spatial position data of the motion unit is collected according to the preset frame rate. In this case, spatial position data is collected only when the position of the motion unit changes, thereby reducing the number of collections and the amount of data stored.

[0058] Figure 3This is a flowchart of a spatiotemporal data processing method in a metaverse scene according to another embodiment of the present invention. A user logs into a metaverse scene provided by the metaverse system through their terminal settings 20. The user can set a spatiotemporal memory function option before entering the scene, or at any time after entering the scene. In this embodiment, the user sets the spatiotemporal memory function option before entering the scene. In the time parameters of this option, the user sets the start and end times, such as [T1, T2]. The terminal device 20 sends the user-selected spatiotemporal memory function option and the set time parameter data to the server. After receiving the data, the server performs the following steps:

[0059] Step S11a, data preprocessing, includes calculating the timing period based on user-set time parameters, obtaining the user's scene ID, and identifying motion units in the current scene. In the metaverse system, the 3D models of objects constituting the scene are labeled with corresponding motion attributes, and the user roles entering the scene also have motion attributes. These motion attributes include, for example, "movable" and "immovable." The motion attribute of the same object can switch between "movable" and "immovable" depending on the specific situation in the scene. For example, the motion attribute of walls, bricks, etc., constituting a building is usually "immovable," but when they are separated from the building by force, their motion attribute changes from "immovable" to "movable." When the motion attribute of each object in the scene is "movable," it is identified as a motion unit. For example, each user role in the space is a motion unit, and each object that can be moved, moved, or have its state changed is a motion unit.

[0060] Step S12a: Obtain spatial data of the metaverse scene. In one embodiment, the spatial data can be stored in a spatiotemporal memory data storage space allocated for the user, and a configuration file can be created to record the user ID, scene ID, start and end times of the spatiotemporal memory data, etc. In another embodiment, since the metaverse system stores spatial data of various scenes, only a configuration file needs to be created, and the user ID, scene ID and its data storage address, start and end times of the spatiotemporal memory data, etc., need to be recorded in the configuration file.

[0061] Step S13a: Obtain the current time information T.

[0062] Step S14a: Compare the current time information T with the start time T1 in the user-set time parameters.

[0063] Step S15a: Determine whether the current time information T is equal to the start time T1 in the user-set time parameters. If the current time information T is equal to the start time T1 in the user-set time parameters, proceed to step S16a. If the current time information does not reach the start time T1 in the user-set time parameters, return to step S13a.

[0064] Step S16a: Synchronously collect the spatial position data of each motion unit according to a preset frame rate, and add a timestamp to the collected data for each frame. The spatial position data includes, for example, the three-dimensional coordinates constituting the motion unit, and may also include other data provided by the metaverse system that can indirectly represent position, such as the height, width, length, speed, or direction of an object. For a regularly shaped object, its model can be simplified, and only the spatial coordinates of the simplified model are collected. For example, for a long, narrow object, only the spatial coordinates of eight points on the outer contour are collected. Similarly, for a user character, only the spatial coordinates of several key body parts are collected, such as the center of the head structure, the torso, and several points on the limbs. When collecting only the spatial data of these simplified models, the original model corresponding to the simplified model must also be identified, and the original model identifier is also used as supplementary data for the position data. There is a data correspondence between the simplified model and the original model. Once the data of the simplified model is known, all the original model data can be obtained according to the data correspondence.

[0065] Step S17a: Store the acquired data for each frame. Specifically, store the acquired data of each frame in chronological order into the spatiotemporal memory data storage space allocated for the user, and record the storage address in the configuration file.

[0066] Step S18a: Monitor whether the current time has reached the termination time. If it has, stop; otherwise, return to step S16a.

[0067] The serialized multi-frame data obtained by storing the data according to the aforementioned steps constitutes visual data of the memory scene with a timeline.

[0068] In the foregoing embodiments, spatial position data of each motion unit in the scene is collected according to a preset frame rate, resulting in comprehensive and continuous data collection. In another embodiment, data is collected only when the position of a motion unit changes, thereby reducing the amount of data. For example, Figure 4 This is a flowchart illustrating the spatial position data processing of each motion unit in a scene when the current time information T is determined to be equal to the starting time T1, according to an embodiment of the present invention.

[0069] Step 1601a: Obtain the current position data of the moving unit in the scene. The position data includes, for example, spatial coordinates, motion parameters including motion speed and direction.

[0070] Step 1602a: Determine whether the current position of each motion unit is changing. For example, when a motion unit has a speed and direction, it can be determined that its position is changing. When the current position of a motion unit is changing, in step S163a, the spatial position data of the motion unit whose position is changing is obtained according to a preset number of frames. Then return. Figure 3 In step S17a, and in step S18a when monitoring the current time T, if the termination time T2 has not yet been reached, return to step S1601a.

[0071] Through the above processing flow, position data is collected only when the position of the moving unit changes, thus reducing the amount of data collected and stored, improving processing efficiency, and reducing the occupation of storage space.

[0072] The metaverse scene provides users with a social space similar to reality. Social interactions between users and between users and the scene, in addition to using text similar to real-world two-dimensional social applications, also include voice and sound. For example, user characters can have voice conversations, and the scene provides various sound effects as needed, such as footsteps, wind, rain, and the different friction sounds of a table rubbing against different floor materials. Therefore, while collecting visual data of the recalled scene, audio data within the scene is also collected. In one embodiment, sound sources in the scene are identified when acquiring spatial trajectory data. When collecting spatial trajectory data of motion units within a preset time period according to the frame rate, the sounds emitted by the sound sources are collected at the sound sources to obtain multiple synchronized audio data. These multiple synchronized audio data and the visual data of the recalled scene constitute the recalled scene data of the first user within the preset time period. In one embodiment, the recalled scene data includes a set of data files. For example, the synchronized audio data of each sound source is stored as a data file, with the sound source identifier as the filename, and the spatial trajectory data of each motion unit is stored as a data file, with the motion unit identifier as the filename. Since the audio data in this embodiment is collected from the sound source, it can realistically recreate the current scene.

[0073] In this embodiment, after obtaining the spatial location data of the user avatar, actions can be further identified based on the spatial location data of the user avatar. During data storage in step S17a, the identified action IDs are serialized in chronological order and stored in the spatiotemporal memory data storage space as chronologically ordered action IDs and their corresponding times. When various actions are defined for the user avatar in the metaverse system, each action has a corresponding identifier, and each action corresponds to a series of intermediate transition actions and spatial location data of the corresponding limbs. This data is stored in the database using the action identifier as an index. For example, for a greeting action, the action lasts for 2 seconds from start to finish. Within this 2-second time period, there are 48 filtered actions and corresponding spatial coordinates of the spatial coordinates of the corresponding limbs (such as hands). Therefore, after obtaining the spatial coordinate data of the spatial coordinates of the user avatar's constituent limbs (such as hands) over a period of time, intermediate transition actions of the limbs are constructed based on the spatial coordinates of each frame. The database is then queried based on these intermediate transition actions to obtain the corresponding action identifier.

[0074] Example 2

[0075] In this embodiment, when the metaverse scene is an AR scene, the spatial data of the metaverse scene includes AR spatial data constructed on the user's end. This AR spatial data includes spatial structure data constructed based on the real-world scene and virtual digital content data superimposed on the real-world scene. For example, it includes environmental map data and corresponding environmental image data constructed by the AR application client based on collected sensor information and environmental image information using technologies such as SLAM (Simultaneous Localization and Mapping) and V-SLAM (Visual Simultaneous Localization and Mapping), as well as virtual icons and virtual characters superimposed on the current scene. In this embodiment, the client sends the aforementioned AR scene's environmental map data, corresponding environmental image data, and superimposed virtual icons and virtual characters to the server. The server refines the scene based on the received data, such as constructing a complete spatial structure based on the environmental map data and image data, including 3D models of houses, streets, squares, and objects. After server-side refinement, the server displays the same scene as the terminal device. In this embodiment, the motion units include, for example, real objects (including people) whose position and state can change in the current scene, the user himself, and virtual digital content.

[0076] In one specific embodiment, the motion unit includes the user's own avatar, the avatars of other users in the scene, and virtual objects whose positions and states can be changed, such as balloons, sky lanterns, etc. (hereinafter, the avatars of other users and virtual objects such as balloons, sky lanterns, etc., are referred to as virtual motion units). In a further embodiment, the user's client also records a video of the current scene and sends the video data of the scene as supplementary data to the server as scene spatial data to help the server improve the scene. When acquiring the spatial trajectory data of the motion unit, the user's own position data, the position data of other users' avatars, and the position data of virtual objects can be provided by the user's client. In order to overlay these avatars and virtual objects into space, the user's client needs to calculate the overlay position, which is the position data.

[0077] Similar to implementation one, when a user selects the time-space recall function option, the time period is set via the time parameter in the option. The client then processes this data for spatiotemporal data processing; see [link to implementation details]. Figure 5 , Figure 5 This is a flowchart of a spatiotemporal data processing method according to another embodiment of the present invention. The method described in this embodiment includes the following steps:

[0078] In step S11b, the client performs data preprocessing, including calculating the timing period based on the user-set time parameter data and reading the user ID.

[0079] In step S12b, the client acquires spatial data of the AR scene. This includes reading environmental map data and corresponding environmental image data, as well as data on virtual markers and virtual characters overlaid on the current scene. This data is then sent to the server.

[0080] In step S13b, the server improves the scene based on the data sent by the client, such as constructing spatial structures, such as three-dimensional models of houses, streets, squares, and objects, based on environmental map data and image data.

[0081] Step S14b: Obtain the current time information T.

[0082] Step S15b: Compare the current time information T with the start time T1 in the user-set time parameters.

[0083] Step S16b determines whether the current time information T is equal to the start time T1 in the user-set time parameters. If the current time information T is equal to the start time T1 in the user-set time parameters, proceed to step S17b. If the current time information does not reach the start time T1 in the user-set time parameters, return to step S14b.

[0084] Step S17b: Determine if there are virtual motion units in the current scene. If there are virtual motion units, then in step S19b, synchronously collect the spatial position data of the user and the virtual motion units at a preset frame rate, and then proceed to step S20b. If there are no virtual motion units in the current scene, then in step S18b, synchronously collect the user's spatial position data at a preset frame rate. The collection time is also recorded when collecting the spatial position data of the user and / or the virtual motion units. When collecting data at a frame rate, each frame is timestamped using the collection time.

[0085] Step S20b: Send the collected spatial location data to the server.

[0086] Step S21b: Store data. The server will serialize and store the received spatial location data of the user and / or virtual motion unit according to time sequence.

[0087] Step S22b: Monitor whether the current time has reached the termination time. If it has, stop; otherwise, return to step S17b.

[0088] In this embodiment, when collecting spatial location data of the user and the virtual motion unit, the scheme of Embodiment 1 can be referred to, and will not be repeated here.

[0089] Example 3

[0090] After a user selects the "Time-Space Recall" function and obtains the time-space data of the recalled scene, the user can set access permissions for that scene, such as specifying which users can access it or the access methods. Similarly, the user can also have access permissions to other users' recalled scenes. When a user gains access to a user's recalled scene and wishes to view it, the user's identity changes to that of an observer. See also Figure 6 , Figure 6 This is a flowchart of a spatiotemporal data processing method in a metaverse scenario according to another embodiment of the present invention. Specifically, it includes the following steps:

[0091] Step S31: Obtain the current location data and viewing angle data of the observing user. The location data includes spatial coordinates and orientation, and the viewing angle data includes spatial angle.

[0092] Step S32: Obtain memory scene data within a preset time period.

[0093] Step S33, data conversion. This involves converting each frame of the recalled scene data into synchronization frame data that needs to be synchronized to the user, based on the user's current location and viewpoint data, and following the timeline of the recalled scene data.

[0094] Step S34: Based on the observed user's current position and perspective, construct a three-dimensional space of the memory scene using the first frame of synchronized frame data.

[0095] Step S35: Calculate the driving parameters of the motion unit in the memory scene based on adjacent synchronized frame data. The driving parameters include motion speed and motion direction.

[0096] Step S36: Change the position or state of the motion unit in the memory scene based on the driving parameters, so that the motion unit changes from the position and / or state of the previous synchronization frame to the position and / or state of the next synchronization frame.

[0097] The server establishes a corresponding spatial coordinate system for each user's memory scene. When an observer selects the option to access a user's memory scene through the client, the client sends an access request to the server, including the owner user ID and scene ID of the scene to be accessed. Based on the access request, the server, after determining that the user has the necessary permissions, sends an access entry address to the user's client. When the observer clicks the entry address, the server sends scene pre-data to the client, causing the observer to enter a blank scene with the same spatial coordinate system as the memory scene. At this time, the observer can determine a position for themselves in this blank scene as their observation position. In step S31, the spatial coordinates, orientation, and spatial angle of the observer's eye position are read in the current blank scene; this spatial angle is used as the visual angle.

[0098] Since a memory scene corresponds to a time period, the user can view the entire time period or select a segment when watching a memory scene. If the user does not specify a viewing time period, the default is the complete time period of the scene from which data needs to be extracted. When the user specifies a time period, that specified time period is used as the time period from which data needs to be extracted. In step S32, memory scene data is obtained according to the determined time period from which data needs to be extracted. When the memory scene data only includes visual data, the visual data of that time period is extracted. When the memory scene data also includes synchronized audio data, the synchronized audio data is separated from the memory scene data and extracted. In one embodiment, when extracting the visual data of the memory scene for that time period, if the scene spatial data and the spatial trajectory data of the motion unit are stored separately, the scene spatial data is first obtained based on the memory scene specified by the user. This includes spatial structure data of the three-dimensional space of the metaverse scene and virtual digital content data in the scene, such as the three-dimensional coordinates, materials, colors, etc., of three-dimensional buildings and three-dimensional object models. Then, based on the determined time period, the spatial trajectory data of the motion unit corresponding to that time period is determined from the time axis of the memory scene data.

[0099] In order to make the reconstructed memory scene match the current observation position of the observation user, in step S33, based on the current position data and viewpoint data of the observation user, each frame of the memory scene data is converted into synchronization frame data that needs to be synchronized to the observation user frame by frame according to the time axis of the memory scene data.

[0100] Since the memory scene data is collected based on the current frame position and viewpoint of the memory scene's user, in one embodiment, when obtaining the first frame data from the memory scene, multiple reference points are selected from the first frame data, and multiple reference lines and reference surfaces are obtained by connecting these multiple reference points. The position and viewpoint at the time of data acquisition are calculated based on the positional relationship between the reference lines / reference surfaces and the spatial coordinate system; these are referred to as the original position and original viewpoint. The original position and original viewpoint are compared with the current position and viewpoint of the observing user to obtain transformation parameter data such as the distance to be translated and the spatial angle of rotation. Then, each frame of the memory scene data is modified frame by frame according to the transformation parameter data, thereby obtaining the synchronization frame data that needs to be synchronized to the observing user.

[0101] In step S34, a three-dimensional space of the memory scene is constructed based on the first frame of synchronized data. The three-dimensional space of the memory scene corresponds to the current position and viewpoint of the observing user. In one embodiment, when the data stored in chronological order includes spatial structure data and spatial trajectory data of motion units, the three-dimensional space of the memory scene is constructed based on the converted first frame of synchronized data, including the spatial structure constituting the scene, the three-dimensional model of virtual objects, and the user avatar, etc. When the spatial structure data and the spatial trajectory data of motion units are stored separately, in one embodiment, a three-dimensional space of the metaverse scene is constructed based on the obtained spatial structure data, which includes virtual digital content data. The overall position, orientation, and spatial angle of the three-dimensional space are changed using the aforementioned calculated conversion parameter data. Then, the motion units are constructed in the three-dimensional space based on the spatial position data of the motion units in the first frame data, thereby obtaining a three-dimensional memory scene visible from the current position and viewpoint of the observing user.

[0102] Starting from the first frame of data, in step S35, the driving parameters of the motion unit in the memory scene are calculated based on subsequent adjacent synchronized frame data. These driving parameters include motion speed and direction. Taking the scene owner user Avatar as an example, the spatial position data of the scene owner user Avatar is obtained from the first and second frame data respectively. The difference in spatial position between the two frames is calculated; this difference is a vector difference, including distance and direction. Then, based on the time difference between the two frames, the motion speed is calculated. Then, in step S36, the position or state of the motion unit in the memory scene is changed based on the driving parameters, so that the motion unit changes from the position and / or state of the first frame to the position and / or state of the second frame. Following this process, the observing user can see the behavior and actions of the scene owner user Avatar in the scene. The processing procedure for other motion units is the same and will not be described further.

[0103] When the spatial trajectory data is the action data of a user character, such as an action sequence composed of multiple action identifiers, the corresponding action identifier and the corresponding time are read from the action sequence. Based on the action identifier, a set of spatial position data for completing the action is obtained. Then, based on the frame rate of the synchronization frame, the time of two adjacent frames is determined. The spatial position data of the two adjacent frames is read from the action spatial position data time period. Then, the speed and direction of the movement are calculated based on the spatial position data of the two adjacent frames.

[0104] The above processing allows the observing user to see the scene where the scene owner is located during the selected time period from their current location. This includes buildings, objects, the user avatar, and events, behaviors, and state changes that occur in the scene. This is also known as a visual scene.

[0105] When the memory scene data also includes synchronized audio data, such as the user avatar's voice data, footsteps, and falling sounds, these synchronized audio data files are separated from the memory scene data filegroup. The sound source identifier is determined based on the synchronized audio data filename, and a sound source matching the identifier is searched in the current scene. The synchronized audio is then played at that sound source. During playback, the audio timestamp is synchronized with the aforementioned visual scene by aligning it with the scene data timestamp.

[0106] The observer can not only see the scene where the scene owner is located during the selected time period, but also hear the sounds in the scene. Since the sounds come from the sound source, it feels more immersive and realistic.

[0107] Example 4

[0108] When the observer views the scene owner's memories as an avatar, because a three-dimensional space has been established, the observer can not only view the memories from any location outside the current scene, but also enter the current scene and even interact with it. See also Figure 7 , Figure 7 This is a flowchart of a data processing method according to another embodiment of the present invention. Figure 6 The processing method includes the following steps:

[0109] Step S41: Monitor whether the position of the observing user in the three-dimensional scene space has changed. If the position of the observing user has changed, this step is no longer executed. Figure 6 Instead of data transformation, step S42 acquires the changed position data, viewpoint data, and first-time information, and then executes step S43. If no changes have occurred, the process continues. Figure 6 In steps S33 to S36, the scene owner user's memory scene is observed at the current location, and in step S41, the location of the observed user is continued to be monitored.

[0110] Step S43: Based on the difference between the observer's reference time system and the memory scene time system and the current first time information, determine the second time information of the memory scene.

[0111] Step S44: Obtain the memory scene data corresponding to the second time information.

[0112] Step S45: Convert the memory scene data corresponding to the second time information into synchronization frame data that needs to be synchronized to the observing user at the current position and perspective.

[0113] Then follow Figure 6 Step 35 calculates the driving parameters of the motion unit in the memory scene based on adjacent synchronized frame data. The driving parameters include motion speed and motion direction. According to step S36, the position of the motion unit in the memory scene is changed based on the driving parameters, so that the motion unit changes from the position and / or state of the previous synchronized frame to the position and / or state of the next synchronized frame.

[0114] Step S46: Determine whether the observation period selected by the user has been reached. If it has been reached, the process ends. If not, return to step S41 and continue to monitor the user's position in the three-dimensional scene space.

[0115] By observing changes in the user's position, the system converts the memory scene data to the user's perspective in real time, thus enabling observation of the memory scene from any angle and any position.

[0116] Example 5

[0117] In another embodiment, secondary recording of the memory scene can also be implemented. For example, when the user is watching the memory scene, each frame of the memory scene data is converted into synchronization frame data that needs to be synchronized to the user, frame by frame according to the timeline of the memory scene data. The synchronization frame data is then recorded, thereby obtaining secondary scene data corresponding to the current position and viewpoint. When recording the synchronization frame data, the spatial position data of the observing user can also be further recorded. In this case, when the observing user enters the memory scene, the recorded secondary scene data includes not only the return scene of the scene's owner but also the observing user, thus achieving the purpose of leaving an observation trace.

[0118] Example 6

[0119] In another embodiment, multiple secondary scene data from different positions and perspectives can be recorded simultaneously. For example, based on a specified position and perspective (hereinafter referred to as the recording camera position), each frame of the recalled scene data is converted into synchronous frame data of multiple specified camera positions frame by frame according to the timeline of the recalled scene data, and the synchronous frame data is recorded, thereby obtaining a secondary recorded video from multiple camera positions.

[0120] The memory scene in this invention can also be an AR scene, that is, a scene composed of spatial structure data constructed from real scene and virtual digital content data superimposed on real scene. The memory scene in this invention can also be a virtual scene, that is, a scene composed of spatial structure data of virtual scene and virtual digital content data in scene. Regardless of the type of scene, spatial structure data constituting the scene will be generated during the recording process. Therefore, the data processing process for showing the memory scene to the observing user is similar to the above processing process, and will not be explained separately here.

[0121] It should be noted that, for clarity, each embodiment of the present invention is described as a combination of a series of actions or processes. Those skilled in the art should understand that the implementation process is not limited by the order of the described actions or processes, and some steps in the embodiments of the present invention may be processed in other orders or simultaneously.

[0122] Example 7

[0123] Corresponding to the method provided by this invention, this invention also provides a spatiotemporal data processing device in a metaverse scenario, see [link to previous article]. Figure 8 , Figure 8This is a schematic block diagram of a first spatiotemporal data processing device in a metaverse scene according to an embodiment of the present invention. The first spatiotemporal data processing device 100 in this embodiment includes a scene spatial data acquisition module 101, a spatial trajectory data acquisition module 102, and a storage module 103. The scene spatial data acquisition module 101 is used to acquire spatial data of the metaverse scene where the first user is located. When the metaverse scene is an AR scene, the spatial data acquired by the scene spatial data acquisition module 101 includes AR spatial data constructed on the user's end, and the AR spatial data includes spatial structure data constructed based on the real scene and virtual digital content data superimposed on the real scene. When the metaverse scene is a virtual scene, the spatial data acquired by the scene spatial data acquisition module 101 includes spatial structure data of the virtual scene and virtual digital content data in the scene.

[0124] The spatial trajectory data acquisition module 102 is connected to the scene spatial data acquisition module 101 to acquire spatial trajectory data of motion units in the metaverse scene within a preset time period. Specifically, the spatial trajectory data acquisition module 102 collects spatial position data of motion units, including the first user, according to preset conditions within the preset time period, and adds a timestamp to each collected data based on reference clock information; it serializes the collected spatial position data multiple times in chronological order to form spatial trajectory data with a time axis. Specifically, when the preset condition is a preset frame rate, the spatial trajectory data acquisition module 102 collects the spatial position data of the motion units according to the preset frame rate. When the preset condition is a position change condition, the spatial trajectory data acquisition module 102 monitors whether the position of the motion units in the scene changes, and in response to a change in the position of the motion units, collects the spatial position data of the motion units according to the preset frame rate. In an optional embodiment, when acquiring the spatial position data of the motion units, the spatial trajectory data acquisition module 102 identifies actions based on the spatial position data of the user or user role; correspondingly, when serializing the collected data multiple times in chronological order, it serializes the identified action IDs in chronological order.

[0125] The storage module 103 is used to store the spatial trajectory data, thereby forming the visual data of the memory scene of the first user within the preset time period.

[0126] In an optional embodiment, the first spatiotemporal data processing device 100 further includes an audio acquisition module 104, used to identify sound sources in the scene and acquire the sounds emitted by the sound sources to obtain multiple synchronized audio data. Correspondingly, the storage module 103 also stores the multiple synchronized audio data.

[0127] The first spatiotemporal data processing device 100 can be implemented by a client or a server. When implemented by a client, the client sends the visual data of the memory scene stored in the storage module 103 to the server for storage.

[0128] Example 8

[0129] Corresponding to the method provided by this invention, this invention also provides another spatiotemporal data processing device in a metaverse scenario, see [link to previous document]. Figure 9 , Figure 9 This is a schematic diagram of a second spatiotemporal data processing device in a metaverse scene according to an embodiment of the present invention. The second spatiotemporal data processing device 200 includes a position data acquisition module 201, a scene data acquisition module 202, a data conversion module 203, a 3D scene construction module 204, a driving parameter module 205, and a motion unit driving module 206. The position data acquisition module 201 acquires the user's current position data and viewpoint data. The position data includes spatial coordinates and orientation, and the viewpoint data includes spatial angles. The scene data acquisition module 202 acquires memory scene data within a preset time period. The data conversion module 203 is connected to the position data acquisition module 201 and the scene data acquisition module 202. Based on the user's current position data and viewpoint data, it converts each frame of the memory scene data into synchronization frame data that needs to be synchronized to the user, frame by frame, according to the timeline of the memory scene data. The 3D scene construction module 204 is connected to the data conversion module 203 and constructs a 3D space of the memory scene based on the user's current position and viewpoint, using the first synchronization frame data. The driving parameter module 205 is connected to the data conversion module 203, and calculates the driving parameters of the motion unit in the memory scene based on adjacent synchronized frame data. The driving parameters include motion speed and motion direction. The motion unit driving module 206 is connected to the driving parameter module 205, and changes the position of the motion unit in the memory scene based on the driving parameters, so that the motion unit changes from the position and / or state of the previous synchronized frame to the position and / or state of the next synchronized frame.

[0130] In one embodiment, the scene data acquisition module 202 acquires the scene spatial data based on the user-specified memory scene; and determines the spatial trajectory data of the motion unit for the corresponding time period from the time axis of the memory scene data based on the user-specified time period.

[0131] When the spatial trajectory data is the spatial position data of a motion unit, the driving parameter module 205 calculates the motion unit parameters in the memory scene based on adjacent synchronized frame data, and then calculates the speed and direction of motion based on the spatial position data of the same motion unit in adjacent synchronized frames. When the spatial trajectory data is the action ID of a user or user character, the driving parameter module 205 obtains the corresponding spatial position data based on the action ID, and then calculates the speed and direction of motion based on the spatial position data of the user or user character in adjacent synchronized frames.

[0132] In an optional embodiment, the second spatiotemporal data processing device 200 further includes a secondary recording module 207 connected to the data conversion module 203. While the data conversion module 203 converts each frame of the recalled scene data into synchronization frame data to be synchronized to the user according to the timeline of the recalled scene data, the secondary recording module 207 records the synchronization frame data to obtain the scene data from the current observer's perspective in the secondary recording. Furthermore, based on multiple camera positions, the secondary recording module 207 can send multiple camera positions to the data conversion module 203. The data conversion module 203 converts each frame of the recalled scene data into synchronization frame data for each of the multiple designated camera positions according to the timeline of the recalled scene data. The secondary recording module 207 records the synchronization frame data corresponding to each camera position, thereby obtaining a secondary recorded video from multiple camera positions.

[0133] Those skilled in the art will understand that the embodiments described herein are preferred embodiments, and the actions, steps, modules, or units involved are not necessarily essential to the embodiments of the present invention. In the above embodiments, the descriptions of each embodiment have their own emphasis; for parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0134] Figure 10 This is a schematic diagram of the structure of an electronic device 60 according to an embodiment of the present invention. The electronic device 60 includes a processor 601, a memory 602, and a communication bus for connecting the processor 601 and the memory 602. The memory 602 stores a computer program that can run on the processor 601. When the processor 601 runs the computer program, it can execute or implement the steps of the methods in the various embodiments of the present invention. The electronic device 60 also includes a communication interface for receiving and sending data. The electronic device 60 can be a server in the embodiments of the present invention, or it can be a terminal device or an AR device in the embodiments of the present invention. Where appropriate, the electronic device can also be referred to as a computing device.

[0135] In some embodiments, processor 601 may be a central processing unit (CPU), graphics processing unit (GPU), application processor (AP), modem processor, image signal processor (ISP), controller, video codec, digital signal processor (DSP), baseband processor, neural-network processing unit (NPU), etc. Processor 601 may also be other general-purpose processors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors may be microprocessors or any conventional processor. The neural network processor (NPU), by drawing inspiration from biological neural network structures, can rapidly process input information and continuously learn itself. Through the NPU electronic device 60, applications such as intelligent cognition, image recognition, face recognition, semantic recognition, speech recognition, and text understanding can be realized.

[0136] In some embodiments, memory 602 may be an internal storage unit of electronic device 60, such as a hard disk or memory of electronic device 60; memory 602 may also be an external storage device of electronic device 60, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on electronic device 600. Memory 602 may include both internal storage units and external storage devices of electronic device 600. Memory 602 can be used to store operating system, application programs, bootloader, data, and other programs, such as program code of computer programs. Memory 602 includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or compact disc read-only memory (CD-ROM). Memory 602 is used to store program code executed by electronic device 60 and data transmitted. Memory 602 can also be used to temporarily store data that has been output or will be output.

[0137] Those skilled in the art will understand that Figure 10 This is merely an example of electronic device 60 and does not constitute a limitation on electronic device 60. Electronic device 60 may include more or fewer components than shown, or combine certain components, or include different components, such as input / output devices, network access devices, etc.

[0138] This invention also provides a computer-readable storage medium, such as the aforementioned memory of various types 602, which stores a computer program or instructions that, when executed, implement the spatiotemporal data processing method in the metaverse scenario involved in the above embodiments.

[0139] This invention also provides a computer program product, including a computer program or instructions, which, when executed, implement the spatiotemporal data processing method in the metaverse scenario described in the above embodiments. For example, the computer program product may be a software installation package.

[0140] Those skilled in the art should understand that the functions of the methods, steps, or related modules / units described in the embodiments of the present invention can be implemented, in whole or in part, by software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product, or by a processor executing computer program instructions. The computer program product includes at least one computer program instruction, which can be composed of corresponding software modules. These software modules can be stored in RAM, flash memory, ROM, EPROM, EEPROM, registers, hard disk, portable hard disk, read-only optical disk (CD-ROM), or any other form of storage medium known in the art. The computer program instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer program instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium, or a semiconductor medium (e.g., SSD).

[0141] Regarding the various devices / products described in the above embodiments, the modules / units included can be software modules / units, hardware modules / units, or a combination of both. For example, for devices / products applied to or integrated into a chip, all of its modules / units can be implemented using hardware methods such as circuits, or at least some modules / units can be implemented using software programs running on a processor integrated within the chip, while the remaining modules / units can be implemented using hardware methods such as circuits. Similarly, for devices / products applied to or integrated into a terminal, all of its modules / units can be implemented using hardware methods such as circuits, or at least some modules / units can be implemented using software programs running on a processor integrated within the terminal, while the remaining modules / units can be implemented using hardware methods such as circuits.

[0142] The above description is merely a specific embodiment of the present invention. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the protection scope of the present invention.

Claims

1. A method for processing spatiotemporal data in a metaverse scenario, characterized in that, include: The system acquires the current location data and viewpoint data of the observing user. The location data includes spatial coordinates and orientation, and the viewpoint data includes spatial angles. Obtain memory scene data within a preset time period; Based on the user's current location data and perspective data, each frame of the memory scene data is converted into synchronization frame data that needs to be synchronized to the user, frame by frame, according to the timeline of the memory scene data. Based on the observed user's current position and perspective, a three-dimensional space of the memory scene is constructed based on the first frame of synchronized frame data; The driving parameters of the motion unit in the memory scene are calculated based on adjacent synchronized frame data. The driving parameters include motion speed and motion direction. as well as Based on the driving parameters, the position of the motion unit in the memory scene is changed so that the position and / or state of the motion unit changes from the position and / or state of the previous synchronization frame to the position and / or state of the next synchronization frame. This includes acquiring memory scene data within a preset time period, including: Obtain spatial data of the metaverse scene where the first user is located; Acquire spatial trajectory data of motion units in the scene within a preset time period; and The spatial trajectory data is stored to form visual data of the first user's memory scene during the preset time period; The acquisition of spatial trajectory data further includes: identifying sound sources in the scene, collecting the sounds emitted by the sound sources at the sound sources to obtain multiple synchronous audio data, and the multiple synchronous audio data and the visual data of the memory scene constitute the memory scene data of the first user within the preset time period.

2. The spatiotemporal data processing method in the metaverse scenario according to claim 1, characterized in that, The steps to obtain spatial trajectory data of moving units in a scene include: Within a preset time period, spatial position data of motion units, including the first user, are collected according to preset conditions, and a timestamp is added to each collected data based on reference clock information; and Spatial location data collected multiple times are serialized in chronological order to form spatial trajectory data with a time axis.

3. The spatiotemporal data processing method in the metaverse scenario according to claim 2, characterized in that, The preset conditions include a preset frame rate, and correspondingly, the spatial position data of the motion unit is collected according to the preset frame rate.

4. The spatiotemporal data processing method in the metaverse scenario according to claim 2, characterized in that, The preset conditions include position changes, and correspondingly further include: monitoring whether the position of the motion unit in the scene changes; in response to the change in the position of the motion unit, collecting the spatial position data of the motion unit according to a preset frame rate.

5. The spatiotemporal data processing method in a metaverse scenario according to any one of claims 2-4, characterized in that, When the motion unit is a user or user role in the scene, obtaining the spatial location data of the motion unit further includes: identifying the action based on the spatial location data of the user or user role. Correspondingly, when serializing the data collected multiple times in chronological order, the identified action ID is serialized in chronological order.

6. The spatiotemporal data processing method in the metaverse scenario according to claim 1, characterized in that, When the metaverse scene is an AR scene, the spatial data of the metaverse scene includes AR spatial data constructed on the user's end. The AR spatial data includes spatial structure data constructed based on the real scene and virtual digital content data superimposed on the real scene.

7. The spatiotemporal data processing method in the metaverse scenario according to claim 1, characterized in that, When the metaverse scene is a virtual scene, the spatial structure data of the metaverse scene includes the spatial structure data of the virtual scene and the virtual digital content data in the scene.

8. A method for processing spatiotemporal data in a metaverse scenario, characterized in that, include: The system acquires the user's current location data and viewpoint data, wherein the location data includes spatial coordinates and orientation, and the viewpoint data includes spatial angles. Obtain memory scene data within a preset time period; Based on the user's current location data and perspective data, each frame of the memory scene data is converted into synchronization frame data that needs to be synchronized to the user, frame by frame, according to the timeline of the memory scene data. Based on the user's current position and perspective, a three-dimensional space of the memory scene is constructed using the first frame of synchronized frame data; The driving parameters of the motion unit in the memory scene are calculated based on adjacent synchronized frame data. The driving parameters include motion speed and motion direction. as well as The position of the motion unit in the memory scene is changed based on the driving parameters, so that the position and / or state of the motion unit changes from the position and / or state of the previous synchronization frame to the position and / or state of the next synchronization frame.

9. The spatiotemporal data processing method in the metaverse scenario according to claim 8, characterized in that, When the recall scene data also includes synchronized audio data, it further includes: Extract one or more synchronized audio data points from the memory scene data; Identify the sound source in the scene corresponding to each synchronized audio data; and The synchronized audio is played at the sound source.

10. The spatiotemporal data processing method in a metaverse scenario according to claim 8 or 9, characterized in that, Further includes: Monitor the user's position in the three-dimensional scene space; In response to a change in the user's location, the system acquires the changed location data, viewpoint data, and first-time information. Based on the difference between the user's reference time system and the memory scene time system, and the current first time information, the second time information of the memory scene is determined; as well as Obtain the memory scene data corresponding to the second time information, and starting from the current second time information, convert each frame of memory scene data into synchronization frame data that needs to be synchronized to the user at the current position and viewpoint.

11. The spatiotemporal data processing method in the metaverse scenario according to claim 8, characterized in that, The steps for obtaining memory scene data within a preset time period further include: Retrieves scene spatial data based on user-specified memory scenarios; and Based on the user-specified time period, the spatial trajectory data of the motion unit corresponding to the time period is determined from the time axis of the memory scene data.

12. The spatiotemporal data processing method in the metaverse scenario according to claim 11, characterized in that, When the spatial trajectory data is the spatial position data of the motion unit, the driving parameters of the motion unit in the memory scene are calculated based on the adjacent synchronized frame data, and the speed and direction of the motion unit are calculated based on the spatial position data of the same motion unit in the adjacent synchronized frames.

13. The spatiotemporal data processing method in the metaverse scenario according to claim 11, characterized in that, When the spatial trajectory data is the action ID of a user or user role, the step of calculating the driving parameters of the motion unit in the memory scene based on adjacent synchronized frame data includes: Obtain the corresponding spatial location data based on the action ID; and The speed and direction of a user or user role are calculated based on spatial location data in adjacent synchronized frames.

14. The spatiotemporal data processing method in the metaverse scenario according to claim 11, characterized in that, Further includes: When converting each frame of the memory scene data into synchronization frame data that needs to be synchronized to the user, the synchronization frame data is recorded.

15. A spatiotemporal data processing device in a metaverse scenario, characterized in that, include: The location data acquisition module is configured to acquire the current location data and viewpoint data of the observing user. The location data includes spatial coordinates and orientation, and the viewpoint data includes spatial angles. The scene data acquisition module is configured to acquire memory scene data within a preset time period; The data conversion module, which is connected to the location data acquisition module and the scene data acquisition module, is configured to convert each frame of the recalled scene data into synchronization frame data that needs to be synchronized to the user, based on the current location data and perspective data of the observing user, and according to the time axis of the recalled scene data. A 3D scene construction module, which is connected to the data conversion module, is configured to construct a 3D space of the memory scene based on the user's current position and perspective, using the first frame of synchronized frame data. A driving parameter module, which is connected to the data conversion module, is configured to calculate the driving parameters of the motion unit in the memory scene based on adjacent synchronized frame data. The driving parameters include motion speed and motion direction. as well as A motion unit driving module, which is connected to the driving parameter module, is configured to change the position of the motion unit in the memory scene based on the driving parameters, so that the motion unit changes from the position and / or state of the previous synchronization frame to the position and / or state of the next synchronization frame. The scene data acquisition module includes: The scene space data acquisition module is configured to acquire the spatial data of the metaverse scene where the first user is located. A spatial trajectory data acquisition module, connected to the scene spatial data acquisition module, is configured to acquire spatial trajectory data of motion units in the metaverse scene within a preset time period; and The storage module is configured to store the spatial trajectory data to form visual data of the first user's memory scene during the preset time period; The spatial trajectory data acquisition module is further configured to identify sound sources in the scene when acquiring spatial trajectory data. It collects the sound emitted by the sound source at the sound source to obtain multiple synchronous audio data. The multiple synchronous audio data and the visual data of the memory scene constitute the memory scene data of the first user within the preset time period.

16. A spatiotemporal data processing device in a metaverse scenario, characterized in that, include: The location data acquisition module is configured to acquire the user's current location data and viewpoint data. The location data includes spatial coordinates and orientation, and the viewpoint data includes spatial angles. The scene data acquisition module is configured to acquire memory scene data within a preset time period; The data conversion module, which is connected to the location data acquisition module and the scene data acquisition module, is configured to convert each frame of the recalled scene data into synchronized frame data that needs to be synchronized to the user, based on the user's current location data and viewpoint data, and according to the timeline of the recalled scene data. A 3D scene construction module, which is connected to the data conversion module, is configured to construct a 3D space of the memory scene based on the user's current position and perspective, using the first frame of synchronized frame data. A driving parameter module, which is connected to the data conversion module, is configured to calculate the driving parameters of the motion unit in the memory scene based on adjacent synchronized frame data. The driving parameters include motion speed and motion direction. as well as A motion unit driving module, which is connected to the driving parameter module, is configured to change the position of the motion unit in the memory scene based on the driving parameters, so that the motion unit changes from the position and / or state of the previous synchronization frame to the position and / or state of the next synchronization frame.

17. An electronic device, characterized in that, It includes a processor and a memory storing computer program instructions, wherein the processor, when executing the computer program instructions, implements the spatiotemporal data processing method as described in any one of claims 1-7, or implements the spatiotemporal data processing method as described in any one of claims 8-14.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the spatiotemporal data processing method as described in any one of claims 1-7, or the spatiotemporal data processing method as described in any one of claims 8-14.

19. A computer program product, characterized in that, It includes computer program instructions that, when executed by a processor, implement the spatiotemporal data processing method as described in any one of claims 1-7, or implement the spatiotemporal data processing method as described in any one of claims 8-14.

Images