Digital twin system based on multi-modal data and construction method

Abstract

The invention discloses a digital twin system based on multi-modal data and a construction method. The construction method comprises the following steps: (1) collecting community data; (2) building a health care model; (3) constructing an auxiliary travel simulation environment; (4) designing a modal fusion and interaction system; and (5) updating and feeding back a mechanism in real time. According to the invention, through comprehensive utilization of multi-modal data, the intelligence and comprehensiveness of the system are enhanced; 2, a personalized health care recommendation system improves the accuracy of health care service; a real-time simulation and feedback mechanism enhances the flexibility and response capability of the system. A personalized health care activity recommendation system is established through a deep learning algorithm, and intelligent and customized health care services can be provided for community residents. In the technical field of auxiliary travel, the system provides a simulation environment for researchers, supports simulation of complex traffic scenes, and promotes innovation and application of technologies such as automatic driving and intelligent traffic.

Description

Technical Field

[0001] The present invention relates to community health care and smart travel technologies, and in particular to a digital twin system based on multimodal data and a construction method thereof. Background Art

[0002] With the acceleration of urbanization and the arrival of an aging society, community health care and smart travel technologies have become key areas for improving the quality of life of residents. Community health care aims to provide community residents with comprehensive health support such as health management, elderly care and life services to meet the health needs of residents of different age groups. Smart travel technologies include autonomous driving and intelligent traffic management, which aim to improve travel efficiency, reduce accident rates and optimize the allocation of traffic resources. The development and application of these two technologies have become an important foundation for building smart cities and improving the quality of life of residents. The introduction of digital twin technology provides a new development direction for the integration of community health care and smart travel. Digital twin technology uses virtualization to perform high-precision digital mapping of physical entities, which can reflect the dynamic changes of the actual environment in real time and predict and optimize through intelligent algorithms. Digital twins can not only be used for virtual modeling in a single field, but also can effectively integrate community health care and smart travel technologies through multimodal data fusion technology, thereby realizing dynamic adjustment of personalized services, real-time simulation and prediction of traffic flow, and collaborative analysis of multi-field data.

[0003] The prior art has the following deficiencies:

[0004] 1. Data fragmentation and information islands: The data sources of the health care system and the transportation system are different. The data sources are diverse and scattered, which makes it difficult to effectively integrate the data. The phenomenon of information islands is serious, which limits the overall performance of the system.

[0005] 2. Insufficient personalized services: Existing community health care services and smart travel systems often lack recognition of individual differences, making it difficult to provide customized services based on residents’ personal health conditions and needs.

[0006] 3. Complexity of the traffic environment: The traffic system changes frequently, especially in dynamic scenarios within the community, and is affected by multiple factors including pedestrians, bicycles, and vehicles. Existing simulation systems are difficult to accurately simulate these complex environments.

[0007] 4. Data privacy and security issues: Community health care and smart travel systems involve a large amount of personal privacy data. Existing technologies have loopholes in data privacy and security protection, which may lead to data leakage and abuse.

[0008] 5. Inadequate user experience: The user interaction interface of existing technologies is usually complex in design, and residents and community managers lack an intuitive and simple experience when operating the system. Summary of the invention

[0009] Purpose of the invention: To address the problems of data silos, insufficient service personalization, inflexible real-time response, etc. in the prior art, the present invention provides a digital twin system based on multimodal data and a construction method thereof.

[0010] Technical solution: The method for constructing a digital twin system based on multimodal data described in the present invention comprises the following steps:

[0011] (1) Community data collection:

[0012] Multimodal perception layer: Use multiple sensors (such as cameras, RFID, LBS, meteorological sensors, etc.) to collect data related to physical space (such as location information of community infrastructure, spatial environment), ecological environment (such as air quality, temperature and humidity, etc.), social space (such as community activities, relationship networks, etc.) and assisted travel (such as traffic flow, road conditions, meteorological data).

[0013] (2) Construction of health care model:

[0014] Model layer: Based on 3DGIS, BIM and other technologies, the health care model of the digital twin system is constructed, which includes the three-dimensional spatial layout of community health care facilities, health care activity recommendation module and community health status real-time monitoring module (such as real-time analysis using health sensor data). The model combines the needs of different groups of people in the community, such as the elderly, the disabled, and students.

[0015] (3) Construction of assisted travel simulation environment:

[0016] Scenario construction layer: Based on BIM (Building Information Model) and traffic simulation model, a simulation environment for assisted travel is created. This simulation environment includes data related to traffic flow, road conditions, and meteorological conditions, allowing technical solutions for assisted travel (such as intelligent transportation, path planning, etc.) to be tested and optimized in simulation.

[0017] (4) Modal fusion and interactive system design:

[0018] Rendering layer: Through low-code engines such as WebGL, UE / Unity, and combined with different modal data, a user-friendly 3D visualization interface is provided to help users intuitively understand health care activities and assisted travel conditions and interact with the system. Users can use this interface to monitor the status of the community, the use of health care facilities, and the effectiveness of assisted travel services.

[0019] (5) Real-time update and feedback mechanism: Data connection layer: Integrate real-time basic data, IoT data (such as IoT sensors and smart device data), and business application data to ensure that the digital twin system can dynamically adapt to the status changes of community health care facilities and assisted travel systems.

[0020] The system builds a digital twin community system by integrating multimodal data within the community, including health-related data (medical resources, residents' health data, community activities, etc.) and assisted travel-related data (traffic flow, weather, road conditions, etc.), to achieve intelligent and real-time management of community health-care and smart travel.

[0021] The present invention integrates multiple data sources to establish a virtual and real synchronized digital twin model, which can monitor the health status of community residents in real time, optimize health care services, simulate and manage community transportation, and provide community managers and residents with more intelligent and comprehensive health care and travel solutions. The key technologies of the system include multimodal data fusion, the application of deep learning algorithms, the construction of a virtual traffic simulation environment, and a data privacy protection mechanism.

[0022] Furthermore, the construction of the health care model includes a personalized health care activity recommendation system based on a deep learning algorithm, specifically:

[0023] (1) A convolutional neural network (CNN) is used to extract features of health care activity images; the personalized health care recommendation system calculates the user's preference weight through the following formula:

[0024] P u =α·F i +β·S u

[0025] Among them, P u represents the health care preference weight of user u, F i The feature vector representing health care activity i, S u represents the historical behavior sequence of user u, and α and β are the adjustable weight parameters of the model.

[0026] (2) Encode the user's historical health care activity records into sequence features;

[0027] (3) Integrate image features with user behavior sequence features to form a recommendation model for generating personalized health and wellness activity recommendations;

[0028] The calculation formula of the health care recommendation model is:

[0029] Pu=α·Fi+β·Su+γ·Eu

[0030] Where Pu represents the health care preference score of resident u, Fi is the feature vector of health care activity i, Su is the behavior data of user u, Eu is the environmental data of user u, α, β, γ are the adjustment coefficients of the model,

[0031] (4) Establish a user feedback mechanism to continuously optimize the recommendation results.

[0032] Furthermore, the assisted travel simulation environment includes the following modules:

[0033] (1) Virtual traffic scene construction module: used to simulate the actual traffic environment of the community, combine the community's traffic facilities, road layout and other information, and generate virtual traffic scenes for system simulation and testing;

[0034] (2) Traffic flow simulation module: Use real-time traffic data to generate traffic flow simulation and analyze problems such as travel peak hours and traffic bottlenecks through simulation technology;

[0035] (3) Intelligent traffic management module: responsible for managing and optimizing the traffic system within the community, using intelligent algorithms to perform traffic signal scheduling and route optimization to improve traffic management efficiency;

[0036] (4) Virtual vehicle and pedestrian behavior simulation module: simulates the behavior of virtual vehicles and pedestrians based on travel data to test the effectiveness and stability of different assisted travel solutions. This module supports virtual simulation of vehicle and pedestrian travel behavior in different scenarios.

[0037] Furthermore, the modal fusion includes different types of multimodal data to fully understand the operating status of community health care and assisted travel systems.

[0038] Furthermore, it includes integrating image data, text data, sequence data and multimodal data.

[0039] Weight distribution formula for multimodal data fusion:

[0040]

[0041] Among them, M is the fused data, D i Represents data of different modes, ω i Represents the weight of the data source, n represents the number of data sources, and this formula is used to dynamically adjust the weights of different data modalities to improve the accuracy of fused data.

[0042] Furthermore, the real-time update and feedback mechanism includes:

[0043] (1) Real-time monitoring and dynamic adjustment of health care services;

[0044] (2) Real-time simulation and optimization feedback of assisted travel technology: Through traffic flow and road status data, the travel conditions in the community can be simulated in real time, and traffic management plans and assisted travel plans can be dynamically optimized and feedback can be provided;

[0045] (3) Data privacy and security mechanism: Use data encryption and access control technologies to protect user privacy, and comply with relevant data privacy laws and regulations to ensure the security of the system.

[0046] The system includes the following main functional modules:

[0047] (1) Multimodal data collection and fusion module: collects and integrates multimodal data such as community residents’ health data, traffic flow data, road condition data, weather data, etc., and provides comprehensive data fusion and analysis.

[0048] (2) Personalized health care service module: Through deep learning technology, a personalized health care activity recommendation system is built based on residents’ health history data and living habits.

[0049] (3) Intelligent Traffic Simulation and Management Module: Provides a virtual traffic simulation environment for the verification and optimization of intelligent travel technologies, such as autonomous vehicles and intelligent traffic signal systems.

[0050] Interactive interface module, used to provide user operation interface and real-time feedback;

[0051] (4) Real-time update and feedback module: Through real-time data update and feedback mechanism, ensure that the system can adjust health care service plans and traffic management strategies in a timely manner.

[0052] (5) Privacy protection and security module: uses multi-level data encryption technology to ensure the privacy and security of user data.

[0053] Compared with the prior art, the present invention has the following beneficial effects:

[0054] 1. Comprehensive integration of multimodal data: The present invention realizes the deep integration of multimodal data in the fields of community health care and smart travel, and improves the system's ability to perceive and understand complex community environments.

[0055] 2. Personalized health care services: Through comprehensive analysis of residents’ health status, behavioral habits and environment, the system can provide personalized health care suggestions for each resident, improving the accuracy and effectiveness of services.

[0056] 3. Intelligent traffic simulation optimization: The virtual traffic simulation environment provided by the system can support the research and application of autonomous driving and intelligent traffic management technologies, accelerating the implementation and development of emerging technologies.

[0057] 4. Real-time feedback and dynamic adjustment: The system has the ability to respond quickly and can dynamically adjust health care and travel services based on the community’s real-time data, ensuring that the system always keeps pace with residents’ needs.

[0058] 5. Data privacy and security: The present invention adopts multi-level encryption technology and privacy protection measures to ensure the security of user data and eliminate users' concerns about data leakage. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 A technology stack for the multimodal digital twin community;

[0060] Figure 2 Recommend global travel interests to patients based on their historical travel records. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be further described below.

[0062] The method for constructing a digital twin system based on multimodal data in this embodiment comprises the following steps:

[0063] 1. System architecture and multimodal data collection

[0064] like Figure 1 As shown, the system architecture of the present invention is divided into a data collection layer, a data fusion layer, a model building layer, and an application layer. The data collection layer is responsible for collecting multimodal data from various sensors, medical equipment, traffic monitoring systems, and other channels within the community. The data includes:

[0065] Health data: Obtain residents’ real-time health status (such as heart rate, blood pressure, etc.) through wearable devices and community medical equipment.

[0066] Travel data: including traffic flow, vehicle speed, road congestion, weather data, etc.

[0067] Environmental data: air quality, temperature, humidity, noise, etc. within the community.

[0068] The data is transmitted to the data fusion layer through wireless sensor networks (WSN), where unified data cleaning and standardization are performed, and a data model is constructed through multimodal data fusion technology. Formula 1: Weight distribution formula for multimodal data fusion:

[0069]

[0070] Among them, M is the fused data, D i Represents data of different modes, ω irepresents the weight of the data source, and n represents the number of data sources. This formula is used to dynamically adjust the weights of different data modalities to improve the accuracy of fused data.

[0071] 2. Construction of personalized health care model

[0072] Based on the data fusion layer, the system uses deep learning technology, especially convolutional neural networks (CNN), to extract features from residents' health data and provide personalized health care recommendations in combination with behavioral data (such as activity records and eating habits). Formula 2: Calculation formula for the health care recommendation model:

[0073] Pu=α·Fi+β·Su+γ·Eu

[0074] Among them, Pu represents the health preference score of resident u, Fi is the characteristic vector of health activity i, Su is the behavioral data of user u, Eu is the environmental data of user u (such as weather, air quality, etc.), and α, β, and y are the adjustment coefficients of the model.

[0075] The personalized health care recommendation system calculates the user's preference weight through the following formula:

[0076] P u =α·F i +β·S u

[0077] Among them, P u represents the health care preference weight of user u, Fi represents the feature vector of health care activity i, s u represents the historical behavior sequence of user u, and α and β are the adjustable weight parameters of the model.

[0078] 3. Intelligent traffic simulation and management

[0079] In order to support the research of intelligent traffic management and autonomous driving technology, the present invention provides a virtual simulation platform. The simulation environment can simulate complex traffic scenes in the community, including the dynamic changes of multiple elements such as roads, intersections, pedestrians, and vehicles.

[0080] 4. Real-time update and feedback mechanism

[0081] The system of the present invention can quickly respond to changes in residents' health conditions or traffic incidents (such as accidents, traffic jams) in the community through a real-time data monitoring and feedback module, and adjust health care services and traffic management strategies.

[0082] For example, when the system detects abnormal health data of a resident, it will automatically recommend appropriate health care activities for him / her and notify relevant medical staff. Correspondingly, the traffic simulation module can adjust the virtual traffic lights in time to deal with emergencies.

[0083] The above is only a preferred embodiment of the present invention and does not limit the present invention in any way. Any technician in the relevant technical field, without departing from the scope of the technical solution of the present invention, makes any form of equivalent replacement or modification to the technical solution and technical content disclosed in the present invention, which does not depart from the content of the technical solution of the present invention and still falls within the protection scope of the present invention.

Claims

1. A method for constructing a digital twin system based on multimodal data, characterized in that: The following steps are involved: (1) Community data collection: Collect data related to community health care and assisted travel; (2) Health care model construction: Based on the needs of community health care, a health care model of the digital twin system is constructed. The model includes the spatial layout of community facilities, a health care activity recommendation system, and a real-time monitoring module for community health status; (3) Construction of assisted mobility simulation environment: Based on BIM and traffic simulation models, a simulation environment for assisted mobility is created to test and optimize assisted mobility technologies; (4) Modal fusion and interactive system design: Through the low-code engine, different modal data are combined to provide users with a three-dimensional visualization interface; (5) Real-time update and feedback mechanism: Based on the analysis model and user interaction feedback, the system data is updated in real time, the system performance is optimized, and real-time service optimization and feedback are provided.

2. The method for constructing a digital twin system based on multimodal data according to claim 1, characterized in that: The construction of the health care model includes a personalized health care activity recommendation system based on a deep learning algorithm, specifically: (1) A convolutional neural network (CNN) is used to extract features of health care activity images; the personalized health care recommendation system calculates the user's preference weight through the following formula: P u =α·F i +β·S u Among them, P u represents the health care preference weight of user u, F i The feature vector representing health care activity i, S u represents the historical behavior sequence of user u, α and β are the adjustable weight parameters of the model, (2) Encode the user's historical health care activity records into sequence features; (3) Integrate image features with user behavior sequence features to form a recommendation model for generating personalized health and wellness activity recommendations; The calculation formula of the health care recommendation model is: Pu=α·Fi+β·Su+γ·Eu Where Pu represents the health care preference score of resident u, Fi is the feature vector of health care activity i, Su is the behavior data of user u, Eu is the environmental data of user u, α, β, γ are the adjustment coefficients of the model, (4) Establish a user feedback mechanism to continuously optimize the recommendation results.

3. The method for constructing a digital twin system based on multimodal data according to claim 1, characterized in that: The assisted travel simulation environment includes the following modules: (1) Virtual traffic scene construction module: used to simulate the actual traffic environment of the community, combine the community's traffic facilities, road layout and other information, and generate virtual traffic scenes for system simulation and testing; (2) Traffic flow simulation module: Use real-time traffic data to generate traffic flow simulation and analyze problems such as travel peak hours and traffic bottlenecks through simulation technology; (3) Intelligent traffic management module: responsible for managing and optimizing the traffic system within the community, using intelligent algorithms to perform traffic signal scheduling and route optimization to improve traffic management efficiency; (4) Virtual vehicle and pedestrian behavior simulation module: This module simulates the behavior of virtual vehicles and pedestrians based on travel data to test the effectiveness and stability of different assisted travel solutions. This module supports virtual simulation of vehicle and pedestrian travel behaviors in different scenarios.

4. The method for constructing a digital twin system based on multimodal data according to claim 1, characterized in that: The modal fusion includes different types of multimodal data to fully understand the operating status of community health care and assisted travel systems.

5. The method for constructing a digital twin system based on multimodal data according to claim 4, characterized in that: Including integrating image data, text data, sequence data and multimodal data, Weight distribution formula for multimodal data fusion: Among them, M is the fused data, D i Represents data of different modes, ω i Represents the weight of the data source, n represents the number of data sources, and this formula is used to dynamically adjust the weights of different data modalities to improve the accuracy of fused data.

6. The method for constructing a digital twin system based on multimodal data according to claim 1, characterized in that: The real-time update and feedback mechanism includes: (1) Real-time monitoring and dynamic adjustment of health care services; (2) Real-time simulation and optimization feedback of assisted travel technology: Through traffic flow and road status data, the travel conditions in the community can be simulated in real time, and traffic management plans and assisted travel plans can be dynamically optimized and feedback can be provided; (3) Data privacy and security mechanism: Use data encryption and access control technologies to protect user privacy, and comply with relevant data privacy laws and regulations to ensure the security of the system.

7. A digital twin system based on multimodal data, characterized in that: include: (1) Data collection module, used to collect multimodal data related to community health care and assisted travel; (2) Health care model module, used to generate personalized health care plans and monitor community health status in real time; (3) Assisted mobility simulation module, used to simulate traffic environments and test assisted mobility technologies; (4) Interactive interface module, used to provide user operation interface and real-time feedback; (5) Real-time update and feedback module, used to dynamically adjust the system's health care services and travel plans.

Images