Management system and method for massive multi-source heterogeneous real-scene three-dimensional data

By using a cross-triggered classification matrix and classification network, the problem of correlation in the management of massive, multi-source, heterogeneous real-world 3D data is solved, achieving efficient data storage and retrieval, and supporting multiple management methods.

CN118779476BActive Publication Date: 2024-11-15ZHICHENG SPACE-TIME (XIAN) INNOVATION TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411245640.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-06
Publication Date
2024-11-15
Estimated Expiration
2044-09-06

AI Technical Summary

Technical Problem

Existing technologies struggle to effectively manage the correlations between massive amounts of heterogeneous, multi-source 3D real-world data, resulting in low data management efficiency.

Method used

By employing a cross-triggered classification matrix and classification network, multiple classifiers are generated through a processor. Data classification and storage are performed using programmatic control and dimensional labeling, and a cross-classification network is established to achieve multi-path data management.

Benefits of technology

It enables efficient storage and retrieval of massive, multi-source, heterogeneous 3D real-world data, and supports various forms of data management, including vertical data hierarchical management and unified management of different production databases.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118779476B_ABST
    Figure CN118779476B_ABST
Patent Text Reader

Abstract

The present invention relates to the technical field of three-dimensional data management, and provides a management system and method for massive multi-source heterogeneous real-scene three-dimensional data, wherein the management system includes: a cross-triggered classification matrix for data type classification, wherein the classification matrix is ​​configured to use a processor as a carrier to generate multiple classifiers for classifying data types through a preset program, and the multiple classifiers constitute a classification network with data cross-classification records; and each classifier is configured to a channel specified by the processor; wherein each channel serves as a transmission channel for data in different databases in a data server; each classifier has a classification model controlled by a program, and the classification model is used to classify the input data to obtain classified data, and the classified data is dimensionally marked.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of processing massive multi-source heterogeneous real-scene three-dimensional data, and in particular to a management system and method for massive multi-source heterogeneous real-scene three-dimensional data. Background Art

[0002] In the process of constructing the metaverse, it is necessary to produce massive multi-source heterogeneous real-scene three-dimensional data. These data include vector data, raster data, tile data, documents, pictures, videos, point clouds, street views, panoramas, etc. When the data are managed in a traditional way, distributed management is adopted. However, it is difficult to define the correlation between data in distributed management. Therefore, the present invention proposes a management system and method for massive multi-source heterogeneous real-scene three-dimensional data. Summary of the invention

[0003] In view of this, the main purpose of the present invention is to provide a management system and method for massive multi-source heterogeneous real-scene three-dimensional data.

[0004] The technical solution adopted by the present invention is as follows:

[0005] A management system for massive multi-source heterogeneous real-scene 3D data, including:

[0006] A cross-triggered classification matrix for data type classification, wherein the classification matrix is ​​configured to generate multiple classifiers for classifying data types through a preset program using a processor as a carrier, wherein the multiple classifiers constitute a classification network with data cross-classification records; and each classifier is configured to a channel specified by the processor; wherein each channel serves as a transmission channel for data in different databases in a data server;

[0007] Each classifier has a classification model controlled by a program, and the classification model is used to classify the input data to obtain classified data, and the classified data is dimensionally labeled;

[0008] The storage server is divided into multiple storage blocks;

[0009] An index writing module is used to determine the data range that can be stored in the storage block according to the classification data, determine the usage dimension of the classification data according to the dimension marking performed on the classification data during the classification process, and generate a storage list of the classification data in the storage block according to the usage dimension;

[0010] The reading index module calls a storage list in a storage block according to data information contained in the search instruction, parses the storage list to obtain corresponding usage dimensions, expands the usage dimensions, and outputs corresponding classification data with the usage dimensions.

[0011] Furthermore, the processor is provided with a classifier generation module, and the classifier generation module has:

[0012] A parameter configuration unit is coupled to a system interface for operating and managing the storage server, and calls a preset program through the system interface and writes parameters and feature vectors into the preset program to generate a primary classifier, wherein the parameters and feature vectors represent classification information of data in a database corresponding to the channel;

[0013] A logic rule configuration unit is coupled to the logic controller and the system interface. It associates primary classifiers with the same parameters and / or feature vectors by calling the logic rules in the logic controller in the system interface, and sets the priority between the associated primary classifiers; then generates multiple classifiers with priority division and mutual association.

[0014] Furthermore, the classifier generation module is also provided with:

[0015] A classification network generation unit, used to generate a classification network with data cross-classification according to the priorities corresponding to the classifiers;

[0016] A recorder is used to link the mutually related classifiers in the classification network in order of priority, and record the classification labels of the data in the classification network in the classifier in order of priority.

[0017] The present application also provides a method for managing massive multi-source heterogeneous real-scene 3D data, using the management system for massive multi-source heterogeneous real-scene 3D data described above, comprising the following steps:

[0018] The data generated in the database of the data server is transmitted to the storage server through the corresponding set channel according to the set period;

[0019] The storage server enables cross-triggering and uses a classification matrix for data type classification to classify the produced data, and records the classification tags obtained by the flow classification process of the data in the classification network by different classifiers in the classification matrix in the order of priority;

[0020] Dimensionally mark the classified data. When storing, determine the usage dimension of the classified data based on the dimension mark performed on the classified data during the classification process, and generate a storage list of the classified data in the storage block based on the usage dimension.

[0021] When reading, a storage list is called in the storage block according to the data information contained in the search instruction, the storage list is parsed to obtain a corresponding usage dimension, the usage dimension is expanded, and the corresponding classification data is output according to the usage dimension;

[0022] Wherein, dimension marking of classified data is to generate dimension information representing the dimension of the data type according to the priority order when the recorder classifies and marks the data in the classifier;

[0023] Expanding the usage dimension refers to expanding the dimension information representing the data type dimension.

[0024] The technology disclosed in this application can classify and store data. When classifying and storing data, the same data body may have multiple uses and the possibility of cross-correlation of data. By cross-classifying such related data and dimensionally marking the classified data, various forms of management can be achieved during use, such as realizing vertical data hierarchical management of warehouse integration technology and management of different production databases. It provides a multi-channel management method for the management of massive multi-source heterogeneous data. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The following drawings are only used to illustrate and explain the present invention, and are not used to limit the scope of the present invention, wherein:

[0026] Figure 1 It is a schematic diagram of the framework principle of the present invention;

[0027] Figure 2 It is a schematic diagram of the framework of the classifier generation module in the present invention. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution, design method and advantages of the present invention clearer, the present invention is further described in detail by specific embodiments in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0029] The multi-source heterogeneous data management system provides comprehensive management capabilities for massive multi-source heterogeneous real-scene 3D original data, achievement data, and business data, ensuring the integrated management and application of multi-source heterogeneous data; the system includes: warehouse management center, data center, distribution center, security center, etc.

[0030] The library management center includes: Original library management: Original library management provides comprehensive management of original library, including functions such as library creation, whole library migration, storage source designation, etc.; Achievement library management: Achievement library management provides comprehensive management of achievement library, including functions such as library creation, whole library migration, library rule setting, storage source designation, etc. Storage source management: Storage source management uniformly manages database physical storage, including functions such as database physical resource addition and database physical resource maintenance; Library rule management: Library rule management sets rules for the database, including multiple types of rule management such as naming rules, layer rules, format rules, and space rules.

[0031] The data center includes: providing various data types involved in project construction, mainly including data warehousing management, data management, data orders, etc.; data warehousing management: different warehousing processes are carried out for data involved in warehousing according to data types, and multi-type and multi-mode (logical error detection, naming specification detection, data attribute field detection) data detection is provided. Data that passes the detection is standardly entered into the warehouse according to the established warehousing process, supporting large file warehousing, breakpoint resumption and other functions. Data management: providing data aggregation management for data involved in warehousing, mainly including data classification, data labeling, data viewing, data preview, data deletion and other functions; data order: providing application capabilities for all data in the system, and each user can apply for the use of data with one click through the collection cart, with the capabilities of data collection, data application, data review, etc.; mainly including data retrieval, data application, data authorization task log and other functions.

[0032] The distribution center provides active distribution, statistics and analysis of various types of data on the platform during project construction, mainly including data distribution, my data, etc.; the system function technical indicators include but are not limited to the following: Data distribution: for various types of data distribution in the data management process, including distribution records, list viewing, data retrieval, security verification, attachment management and other functions; My data: unified management of various types of data distributed to me in the data management process, including data viewing, data downloading, security verification and other functions.

[0033] The Security Center provides security monitoring and tracking of the use of various data resources on the platform during project construction, logs user operations, manages the use permissions of various data resources, and ensures the security of data storage and use. It mainly includes security management, audit management, backup and recovery, etc.; system function technical indicators include but are not limited to the following: Security Management: Provide comprehensive management for data security in the data management process, including department management, permission management, role management, user management, list management, etc. Audit Management: Provides functions such as user log, data log, login log, etc. Backup and Recovery: Supports database backup, version management, database recovery and other functions.

[0034] It can be seen that the data produced daily is diverse, and different data corresponds to different databases. Moreover, the production of the above-mentioned various data does not exist in isolation, but multiple data have interdependent relationships. Therefore, by providing a management method that can use massive data in multiple ways, it is convenient for the later storage, retrieval, use and organization of data.

[0035] In view of this, the present application provides a management system for massive multi-source heterogeneous real-scene three-dimensional data, including: a cross-triggered classification matrix for data type classification, wherein the classification matrix is ​​configured to use a processor as a carrier to generate multiple classifiers for classifying data types through a preset program, and the multiple classifiers constitute a classification network with data cross-classification records; and each classifier is configured to a channel specified by the processor; wherein each channel serves as a transmission channel for data in different databases in a data server;

[0036] Each classifier has a classification model controlled by a program, and the classification model is used to classify the input data to obtain classified data, and the classified data is dimensionally labeled;

[0037] The storage server is divided into multiple storage blocks;

[0038] An index writing module is used to determine the data range that can be stored in the storage block according to the classification data, determine the usage dimension of the classification data according to the dimension marking performed on the classification data during the classification process, and generate a storage list of the classification data in the storage block according to the usage dimension;

[0039] The reading index module calls a storage list in a storage block according to data information contained in the search instruction, parses the storage list to obtain corresponding usage dimensions, expands the usage dimensions, and outputs corresponding classification data with the usage dimensions.

[0040] In the above, a logic controller is provided in the processor, and the logic controller has logic rules for performing logic control on the classifier generation module set according to the channel.

[0041] In the above, the processor is provided with a classifier generation module, and the classifier generation module has: a parameter configuration unit, coupled to a system interface for operating and managing the storage server, calling a preset program through the system interface and writing parameters and feature vectors in the preset program to generate a primary classifier, wherein the parameters and feature vectors represent classification information of data in the database corresponding to the channel;

[0042] A logic rule configuration unit is coupled to the logic controller and the system interface. It associates primary classifiers with the same parameters and / or feature vectors by calling the logic rules in the logic controller in the system interface, and sets the priority between the associated primary classifiers; then generates multiple classifiers with priority division and mutual association.

[0043] In the above, the preset program is:

[0044] from sklearn.neighbors import KNeighborsClassifier

[0045] #Define the KNN classifier parameter N as the number of neighbors;

[0046] knn = KNeighborsClassifier(n_neighbors=10)

[0047] #Training process X is the feature vector, y is the label data / vector;

[0048] knn.fit(X, y);

[0049] In the above, KNN performs classification by measuring the distance between different feature values; its idea is: if most of the k most similar samples (i.e., the nearest ones in the feature space) of a sample in the feature space belong to a certain category, then the sample also belongs to this category.

[0050] In the above, when setting the priority between the associated primary classifiers, the number of parameters and feature vectors in each primary classifier is used as the priority configuration, wherein, when the number of parameters and feature vectors in any two primary classifiers is the same, the corresponding priority is randomly defined.

[0051] In the above, the classifier generation module is also provided with:

[0052] A classification network generation unit, used to generate a classification network with data cross-classification according to the priorities corresponding to the classifiers;

[0053] A recorder is used to link the mutually related classifiers in the classification network in order of priority, and record the classification labels of the data in the classification network in the classifier in order of priority.

[0054] In the above, the processor is provided with a plurality of classifier generation modules, and the plurality of classifier generation modules have the function of being simultaneously coupled to the system interface for simultaneous configuration.

[0055] In the above, the programmed classification model refers to the classifier parameters and feature vectors in the training process through the program.

[0056] In the above, dimension marking of classified data is dimensional information representing the dimension of the data type generated according to the priority order when the recorder classifies and marks the data in the classifier.

[0057] In the above, expanding the usage dimension refers to expanding the dimension information representing the data type dimension.

[0058] The present application also provides a method for managing massive multi-source heterogeneous real-scene 3D data, using the management system for massive multi-source heterogeneous real-scene 3D data described above, comprising the following steps:

[0059] The data generated in the database of the data server is transmitted to the storage server through the corresponding set channel according to the set period;

[0060] The storage server enables cross-triggering and uses a classification matrix for data type classification to classify the produced data, and records the classification tags obtained by the flow classification process of the data in the classification network by different classifiers in the classification matrix in the order of priority;

[0061] Dimensionally mark the classified data. When storing, determine the usage dimension of the classified data based on the dimension mark performed on the classified data during the classification process, and generate a storage list of the classified data in the storage block based on the usage dimension.

[0062] When reading, a storage list is called in the storage block according to the data information contained in the search instruction, the storage list is parsed to obtain a corresponding usage dimension, the usage dimension is expanded, and the corresponding classification data is output according to the usage dimension;

[0063] Wherein, dimension marking of classified data is to generate dimension information representing the dimension of the data type according to the priority order when the recorder classifies and marks the data in the classifier;

[0064] Expanding the usage dimension refers to expanding the dimension information representing the data type dimension.

[0065] The technology disclosed in this application can classify and store data. When classifying and storing data, the same data body may have multiple uses and the possibility of cross-correlation of data. By cross-classifying such related data and dimensionally marking the classified data, various forms of management can be achieved during use, such as realizing vertical data hierarchical management of warehouse integration technology and management of different production databases. It provides a multi-channel management method for the management of massive multi-source heterogeneous data.

[0066] The embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A management system for massive multi-source heterogeneous real-scene 3D data, characterized in that: include: A cross-triggered classification matrix for data type classification, wherein the classification matrix is ​​configured to generate multiple classifiers for classifying data types through a preset program using a processor as a carrier, wherein the multiple classifiers constitute a classification network with data cross-classification records; and each classifier is configured to a channel specified by the processor; wherein each channel serves as a transmission channel for data in different databases in a data server; Each classifier has a classification model controlled by a program, and the classification model is used to classify the input data to obtain classified data, and the classified data is dimensionally labeled; The storage server is divided into multiple storage blocks; An index writing module is used to determine the data range that can be stored in the storage block according to the classification data, determine the usage dimension of the classification data according to the dimension marking performed on the classification data during the classification process, and generate a storage list of the classification data in the storage block according to the usage dimension; The reading index module calls a storage list in a storage block according to data information contained in the search instruction, parses the storage list to obtain corresponding usage dimensions, expands the usage dimensions, and outputs corresponding classification data with the usage dimensions.

2. The management system for massive multi-source heterogeneous real-scene three-dimensional data according to claim 1, characterized in that: The processor is provided with a classifier generation module, and the classifier generation module has: A parameter configuration unit is coupled to a system interface for operating and managing the storage server, and calls a preset program through the system interface and writes parameters and feature vectors into the preset program to generate a primary classifier, wherein the parameters and feature vectors represent classification information of data in a database corresponding to the channel; A logic rule configuration unit is coupled to the logic controller and the system interface. It associates primary classifiers with the same parameters and / or feature vectors by calling the logic rules in the logic controller in the system interface, and sets the priority between the associated primary classifiers; then generates multiple classifiers with priority division and mutual association.

3. The management system for massive multi-source heterogeneous real-scene 3D data according to claim 2, characterized in that: The classifier generation module is also provided with: A classification network generation unit, used to generate a classification network with data cross-classification according to the priorities corresponding to the classifiers; A recorder is used to link the mutually related classifiers in the classification network in order of priority, and record the classification labels of the data in the classification network in the classifier in the order of priority.

4. A method for managing massive multi-source heterogeneous real-scene 3D data, using the management system for massive multi-source heterogeneous real-scene 3D data according to any one of claims 1 to 3, characterized in that: The steps include: The data generated in the database of the data server is transmitted to the storage server through the corresponding set channel according to the set period; The storage server enables cross-triggering and uses a classification matrix for data type classification to classify the produced data, and records the classification tags obtained by the flow classification process of the data in the classification network by different classifiers in the classification matrix in the order of priority; Dimensionally mark the classified data. When storing, determine the usage dimension of the classified data based on the dimension mark performed on the classified data during the classification process, and generate a storage list of the classified data in the storage block based on the usage dimension. When reading, calling a storage list in a storage block according to the data information contained in the search instruction, parsing the storage list to obtain a corresponding usage dimension, expanding the usage dimension, and outputting the corresponding classification data according to the usage dimension; Wherein, dimension marking of classified data is to generate dimension information representing the dimension of the data type according to the priority order when the recorder classifies and marks the data in the classifier; Expanding the usage dimension refers to expanding the dimension information representing the data type dimension.