Method for constructing multi-modal data set in real meta universe environment

Abstract

The invention discloses a method for constructing a multi-modal data set in a real meta-universe environment, which comprises the following steps of: determining the content of the data set, configuring the meta-universe environment, constructing a meta-universe client by using a Unreal Engine engine, and constructing a multi-modal data set in a real meta-universe environment. Collecting image information, text information, current time and user ID of the meta universe user and image information, audio information, text information, current time and scene ID of the scene; the method comprises the following steps of: establishing scenes, users and an interaction set by using MongoDB, creating a document for each scene and each user, and storing image, audio and text information; fusing the data sets on the basis of the user ID, the scene ID and the time; and exporting and sorting the data in the database in a. Csv file form. Through the selection of the meta-universe environment, the dynamic collection of the user information and the configuration at the server and the client, the generation of real and real-time data is ensured, and the constructed data set collection method in the meta-universe environment can be used for researchers to construct the required data set in any scene by themselves.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal dataset construction, and in particular to a method for constructing a multimodal dataset in a metaverse environment. Background Art

[0002] With the rise of the metaverse in recent years, multimodal data in this environment has received more and more attention. Ensuring the validity of multimodal data is very important for specific services in the metaverse. The metaverse environment is different from the cloud environment and the edge environment. Using virtual reality technology, the metaverse environment can provide a more realistic and immersive experience. From the perspectives of sound and image, users can feel the immersive feeling and interact with objects and scenes in the virtual environment, which enhances the user's sense of participation and entertainment experience. However, the variety of service scenarios also makes it difficult for users to make accurate choices. At the same time, the interaction and feedback information has also evolved from clicks and collections on cloud platforms and edge platforms to multiple interactions such as actions, expressions, and sounds. Therefore, how to recommend and provide high-quality targeted services to users in the metaverse environment has attracted the attention of more scholars, and multimodal data sets in the metaverse environment are therefore needed in the research process.

[0003] In terms of multimodal dataset construction, Julian McAuley's team at the University of California constructed the Amazon dataset by crawling product information on the Amazon website and using the metadata and image features provided by Amazon, combined with user behavior data (such as purchase history, ratings, comments, etc.). Jie Tang of Tsinghua University crawled academic search engines and academic databases, collected a large amount of academic paper data and related metadata, and constructed the ASNDS dataset. Qiang Yang of the Hong Kong University of Science and Technology integrated audio data (such as audio features of music) and text data (such as song titles, singer information, etc.) to construct a multimodal music dataset Last.fm. However, the construction methods of the above datasets are mostly in cloud environments or edge environments, and lack the construction methods of datasets in the metaverse environment. When it is necessary to conduct algorithm research on services in the metaverse environment, it is impossible to carry out related work due to the lack of multimodal datasets in this environment. Summary of the invention

[0004] Purpose of the invention: The purpose of the present invention is to overcome the lack of a method for constructing a multimodal dataset in a metaverse environment in the prior art, to provide a basis for researchers to improve the multimodality in a metaverse environment, and to provide a method for constructing a multimodal dataset in a metaverse environment.

[0005] The present invention adopts the following technical solution:

[0006] A method for constructing a multimodal dataset in a metaverse environment comprises the following steps:

[0007] Step (1) determines that the data set contains the following contents: scene ID, service scene text information, service scene image information, service scene audio information, user ID, user text information, user image information, and user scene interaction.

[0008] Step (2) configures the metaverse environment as follows: collect user and service scenario data sets in the metaverse environment.

[0009] Step (3) Use the Unreal Engine virtual engine to build a Metaverse client environment, where the client collects the Metaverse user's image information, text information, current time, user ID, and the scene's image information, audio information, text information, current time, and scene ID;

[0010] Step (4) uses the Tomcat server to collect the user ID, current time, and the evaluation score of the real scene at the current time;

[0011] Step (5) The client sends the data collected in step (3) to the network transceiver buffer area of ​​the server and temporarily stores it;

[0012] Step (6) Use MongoDB to create scenes, users, and interaction collections, create a document for each scene and user, and store images, audio, and text information;

[0013] Step (7) extracting features from scene images, audio, text information, and user images and text information, and fusing the current data set based on user ID, scene ID, and time;

[0014] Step (8) exports and organizes the data in the database in the form of a .csv file.

[0015] Specifically, the service scene text information, service scene image, and service scene audio in step (1) are supporting data collected on a timeline in a metaverse environment, and the above three reflect the multimodal characteristics of the metaverse scene; user text information, user image information, and user interaction data are also user data collected on the same timeline in a metaverse environment, and the above three reflect the multimodal characteristics of metaverse users.

[0016] Specifically, the step (3) comprises:

[0017] Step (31) Configure the Unreal Engine client environment: Use Unreal Engine to create a virtual world project and configure the necessary rendering, physics, and network settings; after the client is loaded, use Blueprint or C++ programming language to integrate image and audio capture components to ensure that user and scene data can be captured in real time in the virtual environment; save the configuration file and preview the effect in the Unreal Engine editor to ensure that all functions are running normally.

[0018] Step (32) collects image and audio information of the user and the scene: in Unreal Engine, the image information of the user and the surrounding environment is captured by a virtual camera, and the audio information is captured by a virtual microphone; RabbitMQ is used to transmit the captured multimedia data in real time; and data compression and optimization algorithms are configured to reduce delays and data loss during network transmission.

[0019] Step (33) Collect text information and user / scenario ID: Allow users to input text information through an interactive user interface (UI) in the virtual environment, and send this information to the server through a network API; assign a unique ID identifier to each user and scene, and capture it together when collecting data to ensure data traceability and consistency.

[0020] Step (34) processes and records the current time information: the client obtains the accurate UTC timestamp through the built-in clock or accesses the Network Time Protocol (NTP) server; the timestamp is converted to local time and packaged with the collected data for time series analysis and real-time monitoring; a time synchronization mechanism is set up to ensure that the time between all clients and servers is consistent to support time-sensitive operations and logging.

[0021] Step (35) Final data integration and transmission: Integrate all collected information (images, audio, text, time, ID) into a unified data format, such as JSON or XML; configure to use a secure transmission protocol, such as TLS / SSL.

[0022] Specifically, the step (4) comprises:

[0023] Step (41) configures the Tomcat server to receive user data: install and configure Apache Tomcat on the server, set appropriate memory and connection parameters to support high concurrent user access; develop a Web application and deploy it on the Tomcat server to receive and process user-submitted data; configure the HTTPS protocol to ensure the security of data transmission.

[0024] Step (42) Develop a user evaluation collection interface: Use Java Servlet technology to develop an API endpoint that allows users to submit their ID, current time, and evaluation of real scenarios through the HTTP POST method; at the data receiving end, use the JSON format to parse the data submitted by the user and verify the integrity and correctness of the data; set up a database connection.

[0025] Step (43) Processing of time information: When the user submits data, the server uses the NTP protocol to synchronize with the Internet time server to ensure that the recorded time information is accurate; the user's timestamp is converted into the local time of the server time zone to facilitate time series related analysis and processing.

[0026] Step (44) Analysis and application of user evaluation data: Design a background processing module to regularly extract user evaluation data from the database and perform statistical analysis; adjust the parameters of the real scene, such as graphics quality, interaction logic, etc., based on user feedback to improve user satisfaction and engagement; use Apache Zeppelin to display the evaluation results and the effects of improvement measures.

[0027] Step (45) Ensure the scalability and maintainability of the system: Use Docker to ensure fast deployment, reliable operation, and easy maintenance of Web applications; configure the Tomcat server's log system to record all key operations and system errors to facilitate problem tracking and performance optimization; perform regular system testing and updates, including stress testing and security vulnerability scanning, to ensure system stability and data security.

[0028] Specifically, the step (5) includes: sending the user ID, current time and evaluation score collected in step 3 in XML format to the network transceiver buffer area of ​​the server side in the form of byte stream through HttpURLConnection to establish an Http connection and temporarily save them.

[0029] Specifically, the step (6) comprises:

[0030] Step (61) Configure MongoDB database: Create a database named InteractiveSystem. Create the following collections in the InteractiveSystem database: Scenes: Store scene data; Users: Store user data; Interactions: Store interaction data between users and scenes.

[0031] Step (62) defines the data model: Scenes contains fields such as scene_id (scene ID), image (image, Hex format), audio (audio, Hex format), description (text description). Users contains fields such as user_id (user ID), profile_image (user image, Hex format), personal_info (text information, such as age, interests, etc.). Interactions contains fields such as scene_id (scene ID), user_id (user ID), interact_period (interaction period), interact_action (interaction score).

[0032] Step (63) Storing multimedia information: For large files (such as high-resolution images and long audio), use MongoDB's GridFS function to store them to avoid the size of a single document exceeding the limit.

[0033] Specifically, the step (7) comprises:

[0034] Step (71) Apply deep learning models for feature extraction: For image feature extraction, choose to use EfficientNet, which is a convolutional network optimized based on the principle of automatic machine learning and has been proven to have efficient performance in various image recognition tasks. For audio feature extraction, you can use YAMNet, which is a model based on deep neural networks that is specifically used for audio event detection and can identify various scenes and object sounds in audio. For text information, use the BERT (Bidirectional Encoder Representations from Transformers) model to extract text features. BERT can understand the deep semantics of language and support a variety of language processing tasks.

[0035] Step (72) Data fusion: Use aggregation pipeline or search operation in MongoDB to fuse user and scene data according to user_id, scene_id, and timestamp to generate new interactive views or documents for further analysis and personalized content services.

[0036] Specifically, the step (8) comprises:

[0037] Step (81) sorts out a data set of size 2000×8 in the order of “scene id, user id, scene image, scene audio, user interaction, user image, user text, interaction time”;

[0038] Step (82) further processes the user scenario interaction by converting the user's implicit behavior data such as dwell time and click count into a quantitative preference index. A score is assigned based on the user's dwell time exceeding a predetermined threshold, and the intensity of the user's expressed interest is quantified based on the user's click frequency.

[0039] Combine explicit ratings and quantified implicit ratings to create a comprehensive user preference rating. Use a weighted method to weight the explicit ratings and the converted implicit ratings according to the preset weights (α, β).

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] The present invention ensures the generation of real and real-time data through dynamic collection of user interaction information in a real metaverse environment and configuration on the virtual engine side and the user side. The constructed data set collection method in a real metaverse environment can enable researchers to construct the required data set by themselves in any scenario. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a framework diagram of the method of the present invention;

[0043] Figure 2 The present invention is a flow chart of the method. DETAILED DESCRIPTION

[0044] The present invention is further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, various equivalent forms of modifications to the present invention by those skilled in the art all fall within the scope defined by the claims attached to this application.

[0045] In the present invention, the Unreal Engine virtual engine, the handle touchpad, the VR glasses and other physical devices are used to collect the multi-modal data required, and the pre-processing, enhancement, annotation and merging are performed to finally generate a complete data set.

[0046] like Figure 1 , 2 , the method mainly includes the following eight steps:

[0047] Step 1: Determine the data set including: scene ID, service scene text information, service scene image information, service scene audio information, user ID, user text information, user image information, and user scene interaction.

[0048] Step 2: Configure the real Metaverse environment to collect the interaction dataset and scene dataset of users using Metaverse services in the real Metaverse environment;

[0049] Step 3: Use the Unreal Engine virtual engine to build the Metaverse client environment. The client collects the Metaverse user's image information, text information, current time, user ID, and the scene's image information, audio information, text information, current time, and scene ID;

[0050] Step (5) The client sends the data collected in step (3) to the network transceiver buffer area of ​​the server and temporarily stores it;

[0051] Step (6) Use MongoDB to create scenes, users, and interaction collections, create a document for each scene and user, and store images, audio, and text information;

[0052] Step (7) extracting features from scene images, audio, text information, and user images and text information, and fusing the current data set based on user ID, scene ID, and time;

[0053] Step (8) exports and organizes the data in the database in the form of a .csv file.

[0054] like Figure 2 As shown, the data set determined in step 1 of the present invention includes: service scene text information, service scene image, service scene audio, user identity text information, user interaction image, and user interaction audio; wherein the service scene text information is used to describe each specific virtual scene; the service scene image and the service scene audio are collected from each scene in the virtual engine; the user identity text information is the user's static data, including number, name, gender, and region; the user interaction image and interaction audio are collected from the interaction process between the user and the virtual engine.

[0055] Step 2 of the present invention configures the real metaverse environment to collect the interaction data set and scene data set of the user using the metaverse service in the real metaverse environment. The specific steps are as follows:

[0056] Step 21: Considering both the Metaverse server and the user, configure the server environment to the Unreal Engine virtual engine, and configure the user environment to the handle, touchpad, and VR glasses.

[0057] Step 3 of the present invention uses the Unreal Engine virtual engine to build a metaverse client environment. The client collects the image information, text information, current time, user ID of the metaverse user and the image information, audio information, text information, current time, and scene ID of the scene. The specific steps are as follows:

[0058] Step 31: Configure the Unreal Engine client environment: Use Unreal Engine to create a virtual world project and configure the necessary rendering, physics, and network settings; after the client is loaded, use Blueprint or C++ programming language to integrate image and audio capture components to ensure that user and scene data can be captured in real time in the virtual environment; save the configuration file and preview the effect in the Unreal Engine editor to ensure that all functions are running normally.

[0059] Step 32: Collect image and audio information of the user and the scene: In Unreal Engine, capture image information of the user and his / her surroundings through a virtual camera, and capture audio information through a virtual microphone; use RabbitMQ to transmit the captured multimedia data in real time; configure data compression and optimization algorithms to reduce latency and data loss during network transmission.

[0060] Step 33: Collect text information and user / scenario IDs: Allow users to enter text information through the interactive user interface (UI) in the virtual environment and send this information to the server through the network API; assign a unique ID identifier to each user and scene and capture it together when collecting data to ensure data traceability and consistency.

[0061] Step 34: Process and record current time information: The client obtains accurate UTC timestamps through the built-in clock or accesses the Network Time Protocol (NTP) server; converts the timestamps to local time and packages them with the collected data for time series analysis and real-time monitoring; sets up a time synchronization mechanism to ensure consistent time between all clients and servers to support time-sensitive operations and logging.

[0062] Step 35: Final data integration and transmission: Integrate all collected information (images, audio, text, time, ID) into a unified data format, such as JSON or XML; configure and use secure transmission protocols, such as TLS / SSL.

[0063] Step (4) of the present invention uses a Tomcat server to collect user IDs, current time, and evaluation scores of real scenes at the current time, specifically including:

[0064] Step 41: Configure Tomcat server to receive user data: Install and configure Apache Tomcat on the server, set appropriate memory and connection parameters to support high concurrent user access; develop a web application and deploy it on the Tomcat server to receive and process user-submitted data; configure the HTTPS protocol to ensure the security of data transmission.

[0065] Step 42: Develop a user evaluation collection interface: Use Java Servlet technology to develop an API endpoint that allows users to submit their ID, current time, and evaluation of real scenarios through the HTTP POST method; at the data receiving end, use JSON format to parse the data submitted by the user and verify the integrity and correctness of the data; set up the database connection.

[0066] Step 43: Processing of time information: When the user submits data, the server uses the NTP protocol to synchronize with the Internet time server to ensure that the recorded time information is accurate; the user's timestamp is converted into the local time of the server time zone to facilitate time series related analysis and processing.

[0067] Step 44: Analysis and application of user evaluation data: Design a background processing module to regularly extract user evaluation data from the database and perform statistical analysis; adjust the parameters of the real scene, such as graphics quality, interaction logic, etc., based on user feedback to improve user satisfaction and engagement; use Apache Zeppelin to display the evaluation results and the effects of improvement measures.

[0068] Step 45: Ensure the scalability and maintainability of the system: Use Docker to ensure fast deployment, reliable operation, and easy maintenance of web applications; configure the Tomcat server's logging system to record all key operations and system errors to facilitate problem tracking and performance optimization; perform regular system testing and updates, including stress testing and security vulnerability scanning, to ensure system stability and data security.

[0069] In step (5) of the present invention, the client sends the data collected in step (3) to the network transceiver buffer area of ​​the server and temporarily stores it, which specifically includes:

[0070] Step 51: The user ID, current time and evaluation score collected in step 3 are sent to the network transceiver buffer area of ​​the server in the form of XML format through HttpURLConnection to establish an Http connection and temporarily save it in the form of byte stream.

[0071] Step (6) of the present invention uses MongoDB to establish scenes, users, and interaction sets, creates a document for each scene and user, and stores images, audio, and text information, specifically including:

[0072] Step 61: Configure MongoDB database: Create a database named InteractiveSystem. Create the following collections in the InteractiveSystem database: Scenes: Store scene data; Users: Store user data; Interactions: Store user-scene interaction data.

[0073] Step 62: Define the data model: The scene set (Scenes) contains fields such as scene_id (scene ID), image (image, Hex format), audio (audio, Hex format), description (text description). The user set (Users) contains fields such as user_id (user ID), profile_image (user image, Hex format), personal_info (text information, such as age, interests, etc.). The interaction set (Interactions) contains fields such as scene_id (scene ID), user_id (user ID), interact_period (interaction period), interact_action (interaction score).

[0074] The data collected by MongoDB are shown in Tables 1, 2, and 3 below (since the image information is too large, only the file header is shown):

[0075] Table 1. Real Metaverse Scenario Table

[0076]

[0077] Table 2 User table

[0078]

[0079] Table 3 Interaction table

[0080]

[0081] Step 63: Storing multimedia information: For large files (such as high-resolution images and long audio), use MongoDB's GridFS function to store them to avoid exceeding the size limit of a single document.

[0082] Step (7) of the present invention extracts features from scene images, audio, text information, and user images and text information, and fuses the current data set based on user ID, scene ID, and time, specifically including:

[0083] Step 71: Apply deep learning models for feature extraction: For image feature extraction, choose to use EfficientNet, which is a convolutional network optimized based on the principles of automatic machine learning and has been proven to have efficient performance in a variety of image recognition tasks. For audio feature extraction, you can use YAMNet, a model based on deep neural networks that is specifically used for audio event detection and can identify a variety of scenes and object sounds in audio. For text information, use the BERT (Bidirectional Encoder Representations from Transformers) model to extract text features. BERT can understand the deep semantics of language and support a variety of language processing tasks.

[0084] Step 72: Data fusion: Use aggregation pipeline or search operation in MongoDB to fuse user and scene data according to user_id and scene_id to generate new interaction data, as shown in Table 4.

[0085] Table 4 User scenario fusion table in real metaverse environment

[0086]

[0087] Step (8) of the present invention exports and organizes the data in the database in the form of a .csv file, specifically including:

[0088] Step 81 sorts out a data set of size 2000×8 in the order of “scene id, user id, scene image, scene audio, user interaction, user image, user text, interaction time”;

[0089] Step 82 further processes the user scenario interaction: the user's implicit behavior data such as dwell time and click count are converted into quantitative preference indicators. A score is assigned based on the user's dwell time exceeding a predetermined threshold, and the intensity of interest expressed by the user is quantified based on the user's click frequency. The explicit score and the quantified implicit score are combined to create a comprehensive user preference score. Using a weighted method, the explicit score and the converted implicit score are weighted and synthesized according to the preset weights (α, β).

[0090] The final data set is shown in Table 5 below (the features are too large to be displayed in compressed form. Image features are selected in order of mean, median, maximum, minimum, and standard deviation; audio features are selected in order of frequency, amplitude, duration, and standard deviation of energy; text features are selected in order of length mean, vocabulary median, word maximum, word minimum, and type label standard deviation):

[0091] Table 5 Multimodal data table under real metaverse environment

[0092]

Claims

1. A method for constructing a multimodal dataset in a metaverse environment, comprising the following steps: Step (1) determines that the data set contains the following contents: scene ID, service scene text information, service scene image information, service scene audio information, user ID, user text information, user image information, and user scene interaction. Step (2) configures the metaverse environment as follows: collect user and service scenario data sets in the metaverse environment. Step (3) Use the Unreal Engine virtual engine to build a Metaverse client environment, where the client collects the Metaverse user's image information, text information, current time, user ID, and the scene's image information, audio information, text information, current time, and scene ID; Step (4) uses the Tomcat server to collect the user ID, current time, and the evaluation score of the real scene at the current time; Step (5) The client sends the data collected in step (3) to the network transceiver buffer area of ​​the server and temporarily stores it; Step (6) Use MongoDB to create scenes, users, and interaction collections, create a document for each scene and user, and store images, audio, and text information; Step (7) extracting features from scene images, audio, text information, and user images and text information, and fusing the current data set based on user ID, scene ID, and time; Step (8) exports and organizes the data in the database in the form of a .csv file.

2. The method for constructing a multimodal dataset in a metaverse environment according to claim 1, characterized in that: The service scene text information, service scene image, and service scene audio in step (1) are supporting data collected on a timeline in the metaverse environment. The above three reflect the multimodal characteristics of the metaverse scene; user text information, user image information, and user interaction data are also user data collected on the same timeline in the metaverse environment. The three are used to evaluate the feedback of the metaverse user group.

3. The method for constructing a multimodal dataset in a metaverse environment according to claim 2, characterized in that: The step (3) specifically comprises: Step (31) Configure the Unreal Engine client environment: Use Unreal Engine to create a virtual world project and configure the necessary rendering, physics, and network settings; after the client is loaded, use Blueprint or C++ programming language to integrate image and audio capture components to ensure that user and scene data can be captured in real time in the virtual environment; save the configuration file and preview the effect in the Unreal Engine editor to ensure that all functions are running normally. Step (32) collects image and audio information of the user and the scene: in Unreal Engine, the image information of the user and the surrounding environment is captured by a virtual camera, and the audio information is captured by a virtual microphone; RabbitMQ is used to transmit the captured multimedia data in real time; and data compression and optimization algorithms are configured to reduce delays and data loss during network transmission. Step (33) Collect text information and user / scenario ID: Allow users to input text information through an interactive user interface (UI) in the virtual environment, and send this information to the server through a network API; assign a unique ID identifier to each user and scene, and capture it together when collecting data to ensure data traceability and consistency. Step (34) processes and records the current time information: the client obtains the accurate UTC timestamp through the built-in clock or accesses the Network Time Protocol (NTP) server; the timestamp is converted to local time and packaged with the collected data for time series analysis and real-time monitoring; a time synchronization mechanism is set up to ensure that the time between all clients and servers is consistent to support time-sensitive operations and logging. Step (35) Final data integration and transmission: Integrate all collected information (images, audio, text, time, ID) into a unified data format, such as JSON or XML; configure to use a secure transmission protocol, such as TLS / SSL.

4. The method for constructing a multimodal dataset in a metaverse environment according to claim 3, characterized in that: The step (4) specifically comprises: Step (41) configures the Tomcat server to receive user data: install and configure Apache Tomcat on the server, set appropriate memory and connection parameters to support high concurrent user access; develop a Web application and deploy it on the Tomcat server to receive and process user submitted data; configure the HTTPS protocol to ensure the security of data transmission. Step (42) Develop a user evaluation collection interface: Use Java Servlet technology to develop an API endpoint that allows users to submit their ID, current time, and evaluation of real scenarios through the HTTP POST method; at the data receiving end, use the JSON format to parse the data submitted by the user and verify the integrity and correctness of the data; set up a database connection. Step (43) Processing of time information: When the user submits data, the server uses the NTP protocol to synchronize with the Internet time server to ensure that the recorded time information is accurate; the user's timestamp is converted into the local time of the server time zone to facilitate time series related analysis and processing. Step (44) Analysis and application of user evaluation data: Design a background processing module to regularly extract user evaluation data from the database and perform statistical analysis; adjust the parameters of the real scene, such as graphics quality, interaction logic, etc., based on user feedback to improve user satisfaction and engagement; use Apache Zeppelin to display the evaluation results and the effects of improvement measures. Step (45) Ensure the scalability and maintainability of the system: Use Docker to ensure fast deployment, reliable operation, and easy maintenance of Web applications; configure the Tomcat server's log system to record all key operations and system errors to facilitate problem tracking and performance optimization; perform regular system testing and updates, including stress testing and security vulnerability scanning, to ensure system stability and data security.

5. The method for constructing a multimodal dataset in a metaverse environment according to claim 4, characterized in that: The step (5) specifically includes: sending the user ID, current time and evaluation score collected in step 3 in XML format to the network transceiver buffer area of ​​the server end in the form of byte stream through HttpURLConnection to establish an Http connection and temporarily save them.

6. The method for constructing a multimodal dataset in a metaverse environment according to claim 5, characterized in that: The step (6) specifically comprises: Step (61) Configure MongoDB database: Create a database named InteractiveSystem. Create the following collections in the InteractiveSystem database: Scenes: Store scene data; Users: Store user data; Interactions: Store interaction data between users and scenes. Step (62) defines the data model: Scenes contains fields such as scene_id (scene ID), image (image, Hex format), audio (audio, Hex format), description (text description). Users contains fields such as user_id (user ID), profile_image (user image, Hex format), personal_info (text information, age, interests, etc.). Interactions contains fields such as scene_id (scene ID), user_id (user ID), interact_period (interaction period), interact_action (interaction score). Step (63) Storing multimedia information: For large files (such as high-resolution images and long audio), use MongoDB's GridFS function to store them to avoid the size of a single document exceeding the limit.

7. The method for constructing a multimodal dataset in a metaverse environment according to claim 6, characterized in that: The step (7) specifically comprises: Step (71) Apply deep learning models for feature extraction: For image feature extraction, choose to use EfficientNet, which is a convolutional network optimized based on the principle of automatic machine learning and has been proven to have efficient performance in various image recognition tasks. For audio feature extraction, you can use YAMNet, which is a model based on deep neural networks that is specifically used for audio event detection and can identify various scenes and object sounds in audio. For text information, use the BERT (Bidirectional Encoder Representations from Transformers) model to extract text features. BERT can understand the deep semantics of language and support a variety of language processing tasks. Step (72) Data fusion: Use aggregation pipeline or search operation in MongoDB to fuse user and scene data according to user_id, scene_id, and timestamp to generate new interactive views or documents for further analysis and personalized content services.

8. The method for constructing a multimodal dataset in a metaverse environment according to claim 7, characterized in that: The step (8) specifically comprises: Step (81) sorts out a data set of size 2000×8 in the order of "scene id, user id, scene image, scene audio, user interaction, user image, user text, interaction time"; Step (82) further processes the user scenario interaction by converting the user's implicit behavior data such as dwell time and click count into a quantitative preference index. A score is assigned based on the user's dwell time exceeding a predetermined threshold, and the intensity of the user's expressed interest is quantified based on the user's click frequency. Combine explicit ratings and quantified implicit ratings to create a comprehensive user preference rating. Use a weighted method to weight the explicit ratings and the converted implicit ratings according to the preset weights (α, β).

Images