Digital twin data processing optimization method and device, computer equipment and storage medium
By acquiring and cleaning digital twin data, using FME for conversion and service release, the problem of low data processing efficiency is solved, efficient data processing and model quality improvement is achieved, and the application of smart cities and industrial manufacturing is supported.
Patent Information
- Application Number
- CN202411046965.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-01
- Publication Date
- 2024-11-12
AI Technical Summary
In the existing digital twin technology, data processing has problems with data source diversity, format compatibility and data quality, resulting in low data processing efficiency and affecting model quality and application value.
By obtaining the relevant sources and types of digital twin data, cleaning, preprocessing and conversion, using FME for data format conversion and service release, realizing the automation and efficiency of data processing.
It realizes rapid conversion of different data formats, improves work efficiency, ensures the data quality and timeliness of the digital twin model, and supports applications in fields such as smart city planning and industrial manufacturing.
Smart Images

Figure CN118939716A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to digital twins, and more specifically to digital twin data processing optimization methods, devices, computer equipment and storage media. Background Art
[0002] Digital twin technology is a revolutionary digital model that achieves deep connection and interaction between the physical world and the digital world by accurately simulating physical entities or systems. In the wave of digital transformation, digital twin technology is considered to be one of the key technologies in promoting smart city planning, optimizing industrial manufacturing, and improving facility management efficiency.
[0003] However, to achieve the ideal goal of digital twin technology, many challenges in data processing must be overcome. The data processing in the existing digital twin application process will have the diversity and complexity of data sources, compatibility issues between different data formats and protocols, and data quality assurance (such as noise, missing values or errors). If the above problems exist in the data processing process, it will directly affect the quality and practicality of the final digital twin model. For example, it is impossible to quickly convert different data formats, and the work efficiency is low, which reduces the actual application effect and value of digital twin technology.
[0004] Therefore, it is necessary to design a new method to quickly convert different data formats and improve work efficiency. Summary of the invention
[0005] The purpose of the present invention is to overcome the defects of the prior art and provide a digital twin data processing optimization method, device, computer equipment and storage medium.
[0006] To achieve the above purpose, the present invention adopts the following technical solution: a digital twin data processing optimization method, comprising:
[0007] Obtain relevant data sources and types of digital twins to obtain initial data and corresponding types;
[0008] Cleaning and preprocessing the initial data according to the type to obtain a processing result;
[0009] Converting the processing result to obtain a conversion result;
[0010] A data processing service is performed according to the conversion result.
[0011] A further technical solution is: the acquisition of the relevant data sources and types of the digital twin to obtain the initial data and the corresponding types includes:
[0012] According to the requirements of the digital twin scenario, relevant data sources and types are obtained from major open source data sources and various commercial data sources to obtain initial data and corresponding types.
[0013] A further technical solution is: the initial data is cleaned and preprocessed according to the type to obtain a processing result, including:
[0014] Cleaning and formatting the data belonging to the basic geographic information type in the initial data to obtain a first processing result;
[0015] Performing noise reduction processing and removing bad lines and strips on the remote sensing type data in the initial data to obtain a second processing result;
[0016] Performing topological check and repair on the data belonging to the basic geographic information type and the data belonging to the CAD in the initial data to obtain a third processing result;
[0017] The processing results include a first processing result, a second processing result, and a third processing result.
[0018] A further technical solution is: converting the processing result to obtain a conversion result includes:
[0019] The processing result is transformed using FME to obtain a transformation result.
[0020] A further technical solution is: using FME to convert the processing result to obtain a conversion result includes:
[0021] Using FME to process the satellite base map, elevation map, and building, road, water body, and vegetation data on the processing results to obtain a basic map processing result;
[0022] Using FME to perform data cleaning, attribute merging, spatial association, standardization and conversion, and field mapping on the processing results to obtain geospatial attribute processing results;
[0023] Use FME to perform data format conversion, model optimization, texture mapping, data integration and adaptation to the digital twin scene on the basic map processing results and the geospatial attribute processing results to convert them into the three-dimensional model required for the digital twin scene to obtain the conversion result.
[0024] A further technical solution is: the data processing service according to the conversion result includes:
[0025] The constructed FME template is used to publish the FME service module using the FME Server platform, and the conversion result is provided as a service.
[0026] The further technical solution is: the FME Server platform is used to publish the constructed FME template to the FME service module, and the conversion result is provided as a service, including:
[0027] Build FME templates and upload them to the FME Server platform;
[0028] Configure the Web service and publish the template combined with the conversion result as an HTTP / HTTPS service;
[0029] Set up scheduled tasks and event triggers;
[0030] Enable multithreading and distributed computing resources;
[0031] Open API interfaces and real-time data stream processing extension functions to dynamically update digital twin models.
[0032] The present invention also provides a digital twin data processing optimization device, comprising:
[0033] A data acquisition unit is used to obtain the relevant data sources and types of the digital twin to obtain the initial data and the corresponding types;
[0034] A processing unit, used for cleaning and preprocessing the initial data according to the type to obtain a processing result;
[0035] A conversion unit, used for converting the processing result to obtain a conversion result;
[0036] A service unit is used to perform data processing services according to the conversion result.
[0037] The present invention further provides a computer device, which includes a memory and a processor. The memory stores a computer program, and the processor implements the above method when executing the computer program.
[0038] The present invention also provides a storage medium, wherein the storage medium stores a computer program, and the computer program implements the above method when executed by a processor.
[0039] The beneficial effect of the present invention compared with the prior art is that the present invention obtains relevant data of the digital twin, cleans, preprocesses and converts it according to its type, and finally performs data processing services to achieve rapid conversion of different data formats and improve work efficiency.
[0040] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.
[0042] Figure 1 A schematic diagram of an application scenario of the digital twin data processing optimization method provided by an embodiment of the present invention;
[0043] Figure 2 A schematic diagram of a process flow of a digital twin data processing optimization method provided by an embodiment of the present invention;
[0044] Figure 3 A schematic block diagram of a digital twin data processing optimization device provided by an embodiment of the present invention;
[0045] Figure 4 A schematic block diagram of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0046] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0047] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.
[0048] It should also be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.
[0049] It should be further understood that the term "and / or" used in the present description and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.
[0050] See also Figure 1 and Figure 2 , Figure 1Schematic diagram of an application scenario of the digital twin data processing optimization method provided in an embodiment of the present invention. Figure 2 A schematic flow chart of a digital twin data processing optimization method provided by an embodiment of the present invention. The digital twin data processing optimization method is applied to a server. The server interacts with the terminal for data, and through steps such as data collection, cleaning and preprocessing, processing conversion and data processing services, it quickly realizes the conversion of different data formats, batch engineering operations, and significantly improves work efficiency. It is suitable for fields such as digital twins, urban planning, land use management and architectural design. By integrating open source and commercial data sources and using FME for data processing and service publishing, it provides a reliable data foundation for the construction of digital twin models, promotes the application of digital twin technology in engineering and urban management, and realizes the improvement of work efficiency and data processing automation.
[0051] Figure 2 Schematic diagram of the process flow of the digital twin data processing optimization method provided by the embodiment of the present invention. Figure 2 As shown, the method includes the following steps S110 to S140.
[0052] S110. Obtain relevant data sources and types of the digital twin to obtain initial data and corresponding types.
[0053] In this embodiment, according to the requirements of the digital twin scenario, relevant data sources and types are obtained from major open source data sources and various commercial data sources to obtain initial data and corresponding types.
[0054] Specifically, clarify the data source and type, and select data with high quality and the latest update time, including basic geographic information data, remote sensing image data, and CAD data.
[0055] The initial data includes satellite basemap and elevation map raster image data, basic geographic information data such as buildings, water bodies, roads, vegetation, POI points of interest, place names and addresses, and CAD engineering design drawing data.
[0056] By utilizing various open source and commercial data sources, we can obtain a rich variety of basic geographic information data, remote sensing image data, and CAD data, laying the foundation for subsequent use in digital twin scenarios.
[0057] By acquiring data from major open source and commercial data sources, you can obtain various types of data, including basic geographic information, remote sensing images, CAD designs, etc., to ensure that the digital twin model has comprehensive and diverse data support. Select data with high data quality and the latest update time to ensure the data quality and timeliness of the digital twin model so that the model can accurately reflect the current actual situation and changes. By integrating open source and commercial data sources and using automated tools such as FME for data processing and conversion, the efficiency of data acquisition, integration and processing is improved, and manual intervention and processing time are reduced. It is suitable for digital twin model construction and multiple fields such as urban planning, land use management and architectural design, providing a reliable data foundation for various engineering and management applications. Through the detailed and real-time data provided by the digital twin model, it supports decision-making and planning optimization, helping to improve the efficiency and quality of urban management.
[0058] S120: Clean and pre-process the initial data according to the type to obtain a processing result.
[0059] In this embodiment, the processing results include a first processing result, a second processing result, and a third processing result.
[0060] Remove invalid and duplicate data and standardize the format to meet the standard format required by digital twins.
[0061] In one embodiment, the above-mentioned step S120 may include steps S121 - S123 .
[0062] S121. Clean and standardize the format of the data belonging to the basic geographic information type in the initial data to obtain a first processing result.
[0063] In this embodiment, invalid data, duplicate data, and erroneous data in the initial data are cleared to ensure the quality and accuracy of the data; the cleaned data is converted into a unified format to meet the standard requirements of the digital twin model.
[0064] Specifically, first, the acquired basic geographic information data is cleaned to remove possible invalid and duplicate data. This includes checking the data integrity, consistency, and legality to ensure that the data quality meets the requirements of digital twin modeling. The cleaned data is standardized to conform to the standard format required by digital twins. This involves unifying the coordinate system, unit standardization, attribute field naming specifications, etc., for subsequent data processing and integration.
[0065] Cleaning and format standardization can effectively improve data quality, make data more reliable and accurate, and help reduce errors and deviations in digital twin models; unified data formats and cleaning processes ensure data consistency, making it easier to manage and analyze; high-quality data is the basis for building accurate digital twin models and can improve the model's prediction and simulation capabilities.
[0066] S122: Perform noise reduction processing and remove bad lines and strips on the remote sensing type data in the initial data to obtain a second processing result.
[0067] In this embodiment, the remote sensing image data is subjected to noise reduction processing to remove the noise introduced by the sensor or the transmission process to improve the data quality and accuracy; the bad lines (such as image break lines caused by sensor failure) and stripes (such as striped noise generated in data transmission) that may exist in the image are detected and eliminated to ensure the continuity and integrity of the image in subsequent processing.
[0068] By reducing noise and removing bad lines, the quality and visualization of remote sensing images can be improved, making them more suitable for the application of digital twin models; processing bad lines and strips can help reduce visual and computational errors in the data and enhance the accuracy and reliability of the model.
[0069] S123, performing topological check and repair on the data belonging to the basic geographic information type and the data belonging to CAD in the initial data to obtain a third processing result.
[0070] In this embodiment, the topological relationship of basic geographic information data and CAD data is checked and repaired, which includes checking the spatial relationship between geometric objects, such as the topological correctness of points, lines, and surfaces, to ensure that the data can correctly express and process spatial relationships when performing spatial analysis and modeling.
[0071] Check the topological relationships in basic geographic information and CAD data, such as nodes, boundaries, and adjacency relationships, to ensure the consistency and integrity of the data; based on the inspection results, repair possible topological errors or inconsistencies in the data.
[0072] Through topology checking and repair, the integrity of basic geographic information and CAD data can be ensured to avoid data inconsistencies and errors in the model; high-quality data topology can improve the precision and accuracy of the digital twin model, making it more suitable for complex simulation and analysis applications.
[0073] Through the above steps, it can be ensured that the data obtained from step S110 has been strictly cleaned, preprocessed and standardized, so that it can be seamlessly used for subsequent digital twin modeling and data integration. These steps not only improve the data quality and consistency, but also lay a solid foundation for the construction of subsequent digital twin models.
[0074] S130: Convert the processing result to obtain a conversion result.
[0075] In this embodiment, the conversion result refers to the result of using FME Workbench to perform coordinate conversion, integration, sharing, distribution and inspection to convert data in different formats into the standard format required by digital twins, including three-dimensional models, geographic information system data, etc.
[0076] Specifically, FME is used to transform the processing result to obtain a transformation result.
[0077] In one embodiment, the above-mentioned step S130 may include steps S131 - S133 .
[0078] S131, using FME to process the satellite base map, elevation map, and building, road, water body, and vegetation data of the processing results to obtain a basic map processing result.
[0079] In this embodiment, the basic map processing result refers to the result obtained by processing the satellite base map, elevation map, and building, road, water body, and vegetation data in the processing result.
[0080] Specifically, the processing flow for satellite basemap and elevation map is as follows:
[0081] Data format conversion and resampling: Convert satellite basemap and elevation map data of different formats and resolutions into a unified format and resolution for subsequent data integration and analysis.
[0082] Coordinate transformation and mosaicking: Transform the coordinate system of geographic data to ensure the spatial consistency and alignment of the data, and simultaneously stitch and mosaic the data to generate continuous geographic information.
[0083] Masking and texturing: Use masking operations to remove unwanted parts and add textures to geometric surfaces to improve the visual quality and realism of your data.
[0084] Web map service slice generation: Slice the processed map data into small images suitable for Web map services to improve the performance of map services and user experience.
[0085] The data processing flow for buildings, roads, water bodies, and vegetation is as follows:
[0086] Topology correction and simplification: Repair and simplify the topological relationship of data to ensure accuracy during data integration.
[0087] Deskew and classification: Correct possible errors or biases in the data and classify the data based on attributes.
[0088] Extraction and merging: Extracting specific objects or features of interest from raw data and merging them into a single dataset.
[0089] Data conversion: Convert the format or coordinate system of data as needed to make it compatible with other datasets.
[0090] In this embodiment, FME can ensure the consistency of the processed geographic data in format, coordinate system and topological relationship, and improve the accuracy and credibility of the data; the automated data processing process reduces the errors and time cost of manual operations, and speeds up the updating and publishing of basic maps; through operations such as masking and texture addition, the map data is made more expressive and visually attractive, enhancing the user's understanding and analysis of geographic information.
[0091] S132. Use FME to perform data cleaning, attribute merging, spatial association, standardization and conversion, and field mapping on the processing results to obtain geospatial attribute processing results.
[0092] In this embodiment, the geospatial attribute processing result refers to the result formed after processing the geospatial attribute data in the processing result. Ensure the accuracy and consistency of the data and provide a reliable data foundation for subsequent digital twin modeling. In this step, use FME to perform the following operations:
[0093] Data cleaning and standardization: Remove invalid data and standardize the format and units of attribute data from different sources to ensure the uniformity and comparability of data.
[0094] Attribute merging and spatial association: Merge the attribute information of different data sources, and effectively associate spatial data with attribute data through spatial association operations to enhance the comprehensive analysis capabilities of the data.
[0095] Field mapping and data conversion: Perform field mapping and data type conversion as required, so that attribute datasets can be seamlessly integrated and applied in the digital twin model.
[0096] In this embodiment, the function of FME ensures that the attribute data maintains consistency and integrity during the integration process, avoiding information gaps and inconsistencies caused by different data sources; through attribute merging and spatial association, the comprehensive utilization of data is improved, supporting more accurate and comprehensive decision-making and planning analysis; the automated processing flow reduces the possibility of human errors and improves the efficiency and reliability of data processing.
[0097] S133. Use FME to perform data format conversion, model optimization, texture mapping, data integration and adaptation to the digital twin scene on the basic map processing results and the geospatial attribute processing results, so as to convert them into the three-dimensional model required for the digital twin scene and obtain the conversion result.
[0098] In this embodiment, FME is used to generate and convert the three-dimensional model of the processed geospatial data to meet the needs of the digital twin scene.
[0099] Specifically, the processed geographic data is converted into a three-dimensional model format suitable for the digital twin scenario, while optimizing the details and effects of the model; texture maps are added to enhance the visual effect of the model, while the attribute data is effectively integrated with the existing three-dimensional model data to enhance the realism and expressiveness of the model.
[0100] FME can process complex geospatial data to ensure the accuracy and realism of the 3D model in terms of geometry and visual effects; the generated 3D model meets the needs of digital twin modeling and supports simulation and analysis based on real geographic data; it effectively integrates attribute data with the 3D model to support the comprehensive application and comprehensive analysis capabilities of the digital twin model.
[0101] Through these detailed steps and operations, FME technology can be used to effectively process basic map data, attribute data, and 3D model data, providing a high-quality, consistent, and reliable data foundation for digital twin modeling.
[0102] S140: Perform data processing services according to the conversion result.
[0103] In this embodiment, the constructed FME template is published to the FME service module using the FME Server platform, and the conversion result is provided as a service. The custom developed FME workflow template is published to the FME service module, and the data conversion and transformation model is published as a service, which has the characteristics of SaaS (Software as a Service), realizes data connection, transformation, automation, and integration into the management system.
[0104] Through automated and efficient data processing, the method of this embodiment is built on the FME Server platform, providing strong support for the application of digital twin technology. The FME Server platform not only realizes the high automation of complex data processing processes, but also builds a data processing architecture specifically for digital twin applications.
[0105] In one embodiment, the above-mentioned step S140 may include steps S141 - S145 .
[0106] S141. Build the FME template and upload it to the FME Server platform.
[0107] In this embodiment, the following process is required when building the FME template:
[0108] Format conversion: By defining the process of data format conversion from different source formats to the target format, we ensure that the data can be correctly interpreted and used, thereby improving the interoperability and usability of the data.
[0109] Coordinate transformation: Set coordinate system transformation parameters to ensure the consistency of geographic data between different coordinate systems, which is particularly important for spatial data analysis in geographic information systems (GIS).
[0110] Data cleaning: Design data cleaning steps to automatically detect and correct erroneous data, improve data quality, and ensure the accuracy and reliability of the twin model.
[0111] Topology check: Set topology rules to perform topology consistency checks to ensure the correctness of spatial data and avoid topological errors that may occur during data processing.
[0112] S142, configuring a Web service, and publishing the template in combination with the conversion result as an HTTP / HTTPS service.
[0113] In this embodiment, the service publishes:
[0114] HTTP / HTTPS protocol: Publish the template as HTTP / HTTPS service, and users can access it at any time through the network, whether on a web page, mobile device or other system.
[0115] Cross-platform access: The configuration service supports access from different platforms, including web pages, mobile devices, and APIs, to meet users' diverse application needs and device choices.
[0116] Real-time response:
[0117] Real-time processing feedback: Provides real-time feedback on task status and processing results, allowing users to promptly understand the progress of data processing and supporting users to make immediate adjustments and optimizations to the data processing process.
[0118] Flexible configuration: allows users to customize data processing parameters and processes according to specific application scenario requirements, providing highly customized data processing solutions.
[0119] Therefore, the method of this embodiment has service accessibility and flexibility.
[0120] S143. Set scheduled tasks and event triggers.
[0121] In this embodiment, the method of this embodiment has an automated scheduling function, specifically, time scheduling: setting scheduled tasks, automatically executing data processing procedures, and ensuring timely updating and processing of digital twin model data.
[0122] Event Scheduling: Configure event triggering mechanisms, such as automatically starting processing tasks when data is updated, to achieve real-time data feedback and processing of digital twin models.
[0123] Conditional Scheduling: Define conditional rules to automatically run data processing tasks when specific conditions are met, improving operational flexibility and efficiency.
[0124] S144. Enable multi-threading and distributed computing resources.
[0125] In this embodiment, the method of this embodiment has the parallel processing function of automated processing, specifically, multi-threaded processing: enabling multi-threading technology to process multiple data tasks simultaneously, significantly improving data processing efficiency, especially in large data volume scenarios.
[0126] Distributed computing: Use distributed computing resources to process large-scale data sets, ensure the efficiency and scalability of the processing process, and adapt to complex data processing needs.
[0127] S145, open API interface and real-time data stream processing extension function to dynamically update the digital twin model.
[0128] In addition, the method of this embodiment also has data security and compliance, integration and scalability, and a SaaS service model.
[0129] Specifically, data security and compliance include:
[0130] Permission control:
[0131] Role and user management: Configure role- and user-based permission control to ensure that only authorized users can access and operate data, thereby ensuring data security and privacy.
[0132] Permission settings: define permission levels for different roles, accurately control data access and operation scope, and meet compliance requirements for data management.
[0133] Data Encryption:
[0134] Transport layer encryption: Use the HTTPS protocol to encrypt data transmission to prevent data from being stolen or tampered with during transmission.
[0135] Storage layer encryption: Configure encryption mechanism for data storage to protect the security of static data and prevent unauthorized access and leakage.
[0136] Integration and scalability include:
[0137] Open API: API interface provision: Release rich API interfaces to support data interaction and function integration with other systems, and promote the comprehensive application and data flow of digital twin models. Secondary development: Provide API documentation and examples to support users to carry out secondary development and function expansion according to specific needs, and enhance the flexibility and adaptability of the system.
[0138] Modular design: Plug-in support: allows new functions to be added through plug-ins and extension modules to cope with changing business needs and expand the functional boundaries of the digital twin platform. Real-time data stream processing: supports the access and processing of real-time data streams, dynamically updates the digital twin model, and maintains the timeliness and practicality of the model.
[0139] SaaS service models include: On-demand service: Subscription model: Provide on-demand subscription data processing services, users do not need to invest in and maintain complex hardware and software infrastructure, reducing usage costs and risks. Resource optimization: Dynamically adjust computing resources according to actual needs to ensure efficient operation of services and optimal resource configuration.
[0140] Elastic expansion: Dynamic resource allocation: Automatically adjust computing resources and service scale according to business needs, and flexibly respond to fluctuations and changes in data processing workload. Expansion support: Provide flexible expansion solutions to support the rapid expansion and upgrade of the digital twin platform to maintain the sustainable development and competitive advantage of the system.
[0141] Through the FME Server platform, the integration of the above functions and features provides comprehensive and reliable support for digital twin data processing. From automated processing capabilities to service accessibility and flexibility, to data security and compliance, as well as integration and scalability, each step strengthens the advantages of digital twin models in real-time, accuracy and flexibility. At the same time, the SaaS service model enables the digital twin platform to dynamically adjust resources and service scale according to user needs, thereby ensuring efficient and stable operation, continuously creating value for users and promoting the process of digital transformation.
[0142] The core advantage of the FME Server platform in digital twin data processing lies in its powerful automation and high customization capabilities. First, by building a templated process, data format conversion, coordinate transformation, data cleaning and topology checking are achieved, ensuring the data accuracy and consistency of the digital twin model. The automated scheduling function enables processing tasks to be automatically executed according to time, events or conditions, thereby ensuring real-time updating and processing of digital twin model data. Parallel processing uses multi-threading and distributed computing technology to achieve efficient processing of large-scale data sets, providing powerful computing support for data processing of digital twin models.
[0143] Secondly, through the accessibility and flexibility of the service, the processing template is published as an HTTP / HTTPS service, and cross-platform access is supported, ensuring the convenience and flexibility of digital twin data processing. In terms of data security and compliance, permission control and data encryption measures ensure the security and compliance of digital twin data. In terms of integration and scalability, FMEServer provides broad space for the functional expansion of the digital twin platform. At the same time, the SaaS service model enables the platform to quickly adjust resources and service scale according to demand and maintain efficient and stable operation.
[0144] The integration of these functions and features provides comprehensive and reliable support for digital twin data processing, enabling it to flexibly respond to changing needs and continuously create value for users.
[0145] In this embodiment, the method is efficient and reliable. Traditional data processing and conversion methods usually require a lot of time for repeated operations and are prone to errors. In contrast, the method of data processing based on FME is intuitive to operate, and users can quickly master the configuration of the workflow. Through the graphical data flow, users only need to drag and drop operations to complete complex and diverse conversion processes, and the workflow can run automatically in the background without human interaction. In addition, FME also has a template feature, and the developed templates can be reused, which greatly reduces the time to configure the workflow, thereby significantly improving the efficiency of data conversion.
[0146] In addition to high efficiency, this method is also flexible and customizable. The FME-based data processing method uses a graphical interface, allowing users to flexibly customize data flows according to their own needs and make necessary adjustments and modifications. This flexibility allows data conversion to be customized according to actual scenarios to meet the needs of different applications. Traditional direct conversion methods usually require a lot of coding to adapt to complex and diverse conversion requirements, which is obviously not as intuitive and efficient as FME.
[0147] In addition, the method also demonstrates excellent ease of use. Data processing based on FME supports the connection and conversion of more than 450 data formats. This means that users can freely convert between different data formats and applications without worrying about format compatibility issues. Traditional methods may only support conversion between limited data formats and applications, limiting the flexibility and operability of data processing. Therefore, FME's rich format support provides users with a wider range of choices and higher operational freedom, making it outstanding in various complex data processing scenarios.
[0148] The above-mentioned digital twin data processing optimization method obtains the relevant data of the digital twin, cleans, preprocesses and converts it according to its type, and finally performs data processing services to achieve rapid conversion of different data formats and improve work efficiency.
[0149] Figure 3 is a schematic block diagram of a digital twin data processing and optimization device provided by an embodiment of the present invention. Figure 3 As shown, corresponding to the above digital twin data processing optimization method, the present invention also provides a digital twin data processing optimization device. The digital twin data processing optimization device 300 includes a unit for executing the above digital twin data processing optimization method, and the device can be configured in a server. Specifically, please refer to Figure 3 The digital twin data processing and optimization device 300 includes a data acquisition unit 301, a processing unit 302, a conversion unit 303 and a service unit 304.
[0150] The data acquisition unit 301 is used to obtain the relevant data source and type of the digital twin to obtain the initial data and the corresponding type; the processing unit 302 is used to clean and preprocess the initial data according to the type to obtain the processing result; the conversion unit 303 is used to convert the processing result to obtain the conversion result; the service unit 304 is used to perform data processing services according to the conversion result.
[0151] In one embodiment, the data acquisition unit 301 is used to acquire relevant data sources and types from major open source data sources and various commercial data sources according to the requirements of the digital twin scenario, so as to obtain initial data and corresponding types.
[0152] In one embodiment, the processing unit 302 includes:
[0153] The first processing subunit is used to clean and standardize the format of the data belonging to the basic geographic information type in the initial data to obtain a first processing result; the second processing subunit is used to perform noise reduction processing, remove bad lines and strips on the remote sensing type data in the initial data to obtain a second processing result; the third processing subunit is used to perform topological inspection and repair on the data belonging to the basic geographic information type and the CAD in the initial data to obtain a third processing result; the processing results include the first processing result, the second processing result and the third processing result.
[0154] In one embodiment, the conversion unit 303 is used to use FME to convert the processing result to obtain a conversion result.
[0155] In one embodiment, the conversion unit 303 includes:
[0156] The map processing subunit is used to use FME to process the satellite base map, elevation map, and building, road, water body, and vegetation data of the processing results to obtain the basic map processing results; the spatial attribute processing subunit is used to use FME to perform data cleaning, attribute merging, spatial association, standardization and conversion, and field mapping of the geospatial attribute data of the processing results to obtain the geospatial attribute processing results; the model conversion subunit is used to use FME to perform data format conversion, model optimization, texture mapping, data integration, and adaptation to the digital twin scene on the basic map processing results and the geospatial attribute processing results to obtain the conversion results.
[0157] In one embodiment, the service unit 304 is used to publish the constructed FME template to the FME service module using the FME Server platform, and provide the conversion result as a service.
[0158] In one embodiment, the service unit 304 includes:
[0159] The template construction subunit is used to build FME templates and upload them to the FME Server platform; the publishing subunit is used to configure Web services and publish the template combined with the conversion results as HTTP / HTTPS services; the setting subunit is used to set scheduled tasks and event triggers; the startup subunit is used to enable multi-threading and distributed computing resources; the opening subunit is used to open API interfaces and real-time data stream processing extension functions to dynamically update the digital twin model.
[0160] It should be noted that technical personnel in the relevant field can clearly understand that the specific implementation process of the above-mentioned digital twin data processing optimization device 300 and each unit can refer to the corresponding description in the aforementioned method embodiment, and for the convenience and conciseness of the description, it will not be repeated here.
[0161] The digital twin data processing and optimization device 300 can be implemented in the form of a computer program. Figure 4 Runs on the computer device shown.
[0162] See also Figure 4 , Figure 4 5 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device 500 may be a server, wherein the server may be an independent server or a server cluster composed of multiple servers.
[0163] See also Figure 4The computer device 500 includes a processor 502 , a memory and a network interface 505 connected via a system bus 501 , wherein the memory may include a non-volatile storage medium 503 and an internal memory 504 .
[0164] The non-volatile storage medium 503 can store an operating system 5031 and a computer program 5032. The computer program 5032 includes program instructions, which, when executed, can enable the processor 502 to execute a digital twin data processing optimization method.
[0165] The processor 502 is used to provide computing and control capabilities to support the operation of the entire computer device 500 .
[0166] The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute a digital twin data processing optimization method.
[0167] The network interface 505 is used to communicate with other devices over the network. Figure 4 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present application, and does not constitute a limitation on the computer device 500 to which the solution of the present application is applied. The specific computer device 500 may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0168] The processor 502 is used to run the computer program 5032 stored in the memory to implement the following steps:
[0169] Obtain relevant data sources and types of digital twins to obtain initial data and corresponding types; clean and preprocess the initial data according to the types to obtain processing results; convert the processing results to obtain conversion results; and perform data processing services based on the conversion results.
[0170] In one embodiment, the processor 502 implements the following steps when implementing the step of obtaining the relevant data source and type of the digital twin to obtain the initial data and the corresponding type:
[0171] According to the requirements of the digital twin scenario, relevant data sources and types are obtained from major open source data sources and various commercial data sources to obtain initial data and corresponding types.
[0172] In one embodiment, when the processor 502 implements the step of cleaning and preprocessing the initial data according to the type to obtain a processing result, the processor 502 specifically implements the following steps:
[0173] The data belonging to the basic geographic information type in the initial data are cleaned and format standardized to obtain a first processing result; the data belonging to the remote sensing type in the initial data are subjected to noise reduction processing, bad lines and strips are removed to obtain a second processing result; the data belonging to the basic geographic information type and the CAD in the initial data are subjected to topological inspection and repair to obtain a third processing result;
[0174] The processing results include a first processing result, a second processing result, and a third processing result.
[0175] In one embodiment, when the processor 502 implements the step of converting the processing result to obtain the conversion result, the processor 502 specifically implements the following steps:
[0176] The processing result is transformed using FME to obtain a transformation result.
[0177] In one embodiment, when the processor 502 implements the step of using FME to transform the processing result to obtain the transformation result, the processor 502 specifically implements the following steps:
[0178] Use FME to process the satellite base map, elevation map, and building, road, water body, and vegetation data of the processing results to obtain the basic map processing results; use FME to perform data cleaning, attribute merging, spatial association, standardization and conversion, and field mapping of the geospatial attribute data of the processing results to obtain the geospatial attribute processing results; use FME to perform data format conversion, model optimization, texture mapping, data integration, and adaptation to the digital twin scene on the basic map processing results and the geospatial attribute processing results to convert them into the three-dimensional model required for the digital twin scene to obtain the conversion result.
[0179] In one embodiment, when the processor 502 implements the step of performing data processing service according to the conversion result, the processor 502 specifically implements the following steps:
[0180] The constructed FME template is used to publish the FME service module using the FME Server platform, and the conversion result is provided as a service.
[0181] In one embodiment, the processor 502 implements the following steps when implementing the step of publishing the constructed FME template to the FME service module using the FME Server platform and providing the conversion result as a service:
[0182] Build FME templates and upload them to the FME Server platform; configure Web services and publish the templates combined with the conversion results as HTTP / HTTPS services; set up scheduled tasks and event triggers; enable multi-threading and distributed computing resources; open API interfaces and real-time data stream processing extensions to dynamically update digital twin models.
[0183] It should be understood that in the embodiment of the present application, the processor 502 may be a central processing unit 302 (CPU), and the processor 502 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.
[0184] It can be understood by those skilled in the art that all or part of the processes in the method for implementing the above embodiment can be completed by instructing the relevant hardware through a computer program. The computer program includes program instructions, and the computer program can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiment of the above method.
[0185] Therefore, the present invention also provides a storage medium. The storage medium may be a computer-readable storage medium. The storage medium stores a computer program, wherein when the computer program is executed by a processor, the processor executes the following steps:
[0186] Obtain relevant data sources and types of digital twins to obtain initial data and corresponding types; clean and preprocess the initial data according to the types to obtain processing results; convert the processing results to obtain conversion results; and perform data processing services based on the conversion results.
[0187] In one embodiment, when the processor executes the computer program to implement the step of obtaining the relevant data source and type of the digital twin to obtain the initial data and the corresponding type, the processor specifically implements the following steps:
[0188] According to the requirements of the digital twin scenario, relevant data sources and types are obtained from major open source data sources and various commercial data sources to obtain initial data and corresponding types.
[0189] In one embodiment, when the processor executes the computer program to implement the step of cleaning and preprocessing the initial data according to the type to obtain a processing result, the processor specifically implements the following steps:
[0190] The data belonging to the basic geographic information type in the initial data are cleaned and format standardized to obtain a first processing result; the data belonging to the remote sensing type in the initial data are subjected to noise reduction processing, bad lines and strips are removed to obtain a second processing result; the data belonging to the basic geographic information type and the CAD in the initial data are subjected to topological inspection and repair to obtain a third processing result;
[0191] The processing results include a first processing result, a second processing result, and a third processing result.
[0192] In one embodiment, when the processor executes the computer program to implement the step of converting the processing result to obtain the conversion result, the processor specifically implements the following steps:
[0193] The processing result is transformed using FME to obtain a transformation result.
[0194] In one embodiment, when the processor executes the computer program to implement the step of using FME to convert the processing result to obtain the conversion result, the processor specifically implements the following steps:
[0195] Use FME to process the satellite base map, elevation map, and building, road, water body, and vegetation data of the processing results to obtain the basic map processing results; use FME to perform data cleaning, attribute merging, spatial association, standardization and conversion, and field mapping of the geospatial attribute data of the processing results to obtain the geospatial attribute processing results; use FME to perform data format conversion, model optimization, texture mapping, data integration, and adaptation to the digital twin scene on the basic map processing results and the geospatial attribute processing results to convert them into the three-dimensional model required for the digital twin scene to obtain the conversion result.
[0196] In one embodiment, when the processor executes the computer program to implement the step of performing data processing services according to the conversion result, the processor specifically implements the following steps:
[0197] The constructed FME template is used to publish the FME service module using the FME Server platform, and the conversion result is provided as a service.
[0198] In one embodiment, when the processor executes the computer program to implement the step of publishing the constructed FME template to the FME service module using the FME Server platform and providing the conversion result as a service, the following steps are specifically implemented:
[0199] Build FME templates and upload them to the FME Server platform; configure Web services and publish the templates combined with the conversion results as HTTP / HTTPS services; set up scheduled tasks and event triggers; enable multi-threading and distributed computing resources; open API interfaces and real-time data stream processing extensions to dynamically update digital twin models.
[0200] The storage medium may be a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk, or an optical disk, etc., which are computer-readable storage media that can store program codes.
[0201] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.
[0202] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of each unit is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.
[0203] The steps in the method of the embodiment of the present invention can be adjusted in order, combined and deleted according to actual needs. The units in the device of the embodiment of the present invention can be combined, divided and deleted according to actual needs. In addition, the functional units in various embodiments of the present invention can be integrated into a processing unit 302, or each unit can exist physically separately, or two or more units can be integrated into one unit.
[0204] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, terminal, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention.
[0205] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.
Claims
1. A digital twin data processing optimization method, characterized in that: include: Obtain relevant data sources and types of digital twins to obtain initial data and corresponding types; Cleaning and preprocessing the initial data according to the type to obtain a processing result; Converting the processing result to obtain a conversion result; A data processing service is performed according to the conversion result.
2. The digital twin data processing optimization method according to claim 1 is characterized in that: The acquisition of the relevant data sources and types of the digital twin to obtain the initial data and the corresponding types includes: According to the requirements of the digital twin scenario, relevant data sources and types are obtained from major open source data sources and various commercial data sources to obtain initial data and corresponding types.
3. The digital twin data processing optimization method according to claim 1 is characterized in that: The cleaning and preprocessing of the initial data according to the type to obtain a processing result includes: Cleaning and formatting the data belonging to the basic geographic information type in the initial data to obtain a first processing result; Performing noise reduction processing and removing bad lines and strips on the remote sensing type data in the initial data to obtain a second processing result; Performing topological check and repair on the data belonging to the basic geographic information type and the data belonging to the CAD in the initial data to obtain a third processing result; The processing results include a first processing result, a second processing result, and a third processing result.
4. The digital twin data processing optimization method according to claim 1, characterized in that: The converting the processing result to obtain a conversion result includes: The processing result is transformed using FME to obtain a transformation result.
5. The digital twin data processing optimization method according to claim 4 is characterized in that: The using FME to transform the processing result to obtain the transformation result includes: Using FME to process the satellite base map, elevation map, and building, road, water body, and vegetation data on the processing results to obtain a basic map processing result; Using FME to perform data cleaning, attribute merging, spatial association, standardization and conversion, and field mapping on the processing results to obtain geospatial attribute processing results; Use FME to perform data format conversion, model optimization, texture mapping, data integration and adaptation to the digital twin scene on the basic map processing results and the geospatial attribute processing results to convert them into the three-dimensional model required for the digital twin scene to obtain the conversion result.
6. The digital twin data processing optimization method according to claim 1, characterized in that: The data processing service according to the conversion result includes: The constructed FME template is used to publish the FME service module using the FME Server platform, and the conversion result is provided as a service.
7. The digital twin data processing optimization method according to claim 6, characterized in that: The FME Server platform is used to publish the constructed FME template to the FME service module, and the conversion result is provided as a service, including: Build FME templates and upload them to the FME Server platform; Configure the Web service and publish the template combined with the conversion result as an HTTP / HTTPS service; Set up scheduled tasks and event triggers; Enable multithreading and distributed computing resources; Open API interfaces and real-time data stream processing extension functions to dynamically update digital twin models.
8. A digital twin data processing and optimization device, characterized in that: include: A data acquisition unit is used to obtain the relevant data sources and types of the digital twin to obtain the initial data and the corresponding types; A processing unit, used for cleaning and preprocessing the initial data according to the type to obtain a processing result; A conversion unit, used for converting the processing result to obtain a conversion result; A service unit is used to perform data processing services according to the conversion result.
9. A computer device, characterized in that: The computer device comprises a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.
10. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.
Citation Information
Cited By
Adaptive TIN digital twin terrain grid density adjustment method and device based on machine learning
CN119229060A
