Conversion method and system based on multi-source data

Abstract

The invention relates to the technical field of data processing, and particularly provides a multi-source data-based conversion method and system, comprising: acquiring original data from a plurality of data sources, the data sources including but not limited to a structured data source, a semi-structured data source and an unstructured data source; performing format unification on the cleaned data according to a preset standardization rule, wherein the standardization rule comprises a coding format, field mapping and data type conversion; carrying out encryption processing on the standardized data, and setting a data access permission to ensure the security and privacy of the data; the standardized multi-source data are fused, redundant information is eliminated, and consistency and integrity of the data are guaranteed; storing the fused data into a target database, and supporting dynamic expansion and data version management; standardized data services are provided for external applications through an API interface, and real-time query and analysis of data are supported. The method has the effects of protecting the data security and encrypting the data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a conversion method and system based on multi-source data. BACKGROUND

[0002] In today's digital age, the importance of data is increasingly prominent, and various data resources have become an important basis for decision-making by enterprises and institutions. However, enterprises often face problems such as multiple data sources, non-uniform data formats, and uneven data quality in practical applications, which brings challenges to the practical application of data. Therefore, it is particularly urgent to develop a conversion method that can effectively integrate multi-source data.

[0003] Current technical solutions mainly focus on the processing of single data sources, and there are deficiencies in the integration and conversion of multi-source data. Specifically, data from different sources may use different encoding, formats, and storage methods. Direct integration of these data not only consumes time and effort, but also may result in unusable data or redundant information due to inconsistent formats, affecting the accuracy and usability of the data. Therefore, there is an urgent need for a technical solution that can efficiently process various source data, unify data formats, improve data quality, and effectively integrate multi-source data to meet the needs of modern informationization and intelligent management.

[0004] Prior art one, Chinese patent, application number CN202510511404.5 discloses a scientific and technological achievement transformation potential prediction method and system based on multi-source data fusion. It judges whether the similarity between a certain abstract data of a scientific and technological achievement text data and a certain data label of a text database is greater than a first threshold value. If it is not greater than the first threshold value, at least one target keyword is selected in the abstract data according to a pre-set selection strategy, and a first transformation potential value between at least one first target enterprise data and the scientific and technological achievement text data and a second transformation potential value between a second target enterprise data and the scientific and technological achievement text data are calculated. Based on a pre-set fusion strategy, the first transformation potential value and the second transformation potential value are fused to obtain the transformation potential value of the scientific and technological achievement text data. It solves the problem of inaccurate matching results caused by the deviation between the data label of the scientific and technological achievement data and the actual content of the scientific and technological achievement data;

[0005] Prior art two, Chinese patent, application number CN202111003098.2 discloses a method for realizing the conversion of multi-source heterogeneous data to FHIR standard, relating to the technical field of data conversion, the scheme includes: defining a general data FHIR standardization rule; the FHIR standard library receives data from different types of databases and different structures; based on the defined FHIR standardization rule, the data received by the FHIR standard library is governed into FHIR standard format; encapsulate data security rules to realize data security, privacy protection and permission control, at the same time, define the data interaction mode, open the data through FHIR API and provide FHIR application service. The present application also discloses a system for realizing the conversion of multi-source heterogeneous data to FHIR standard, which realizes the foregoing method through a rule definition module, a data receiving module, a data processing module and a data opening module. The present application can improve the data standardization efficiency through batch standardization of data, and provides effective support for data interconnection;

[0006] The prior art one and the prior art two do not involve data security issues, may cause data leakage risk, cannot adapt to changes in data sources and formats, lack flexibility and scalability, and increase maintenance costs. Therefore, the present application provides a conversion method and system based on multi-source data. SUMMARY

[0007] To achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0008] In one aspect of the present application, a conversion method based on multi-source data is provided, comprising the following steps:

[0009] Obtain raw data from multiple data sources, including but not limited to structured data sources, semi-structured data sources and unstructured data sources;

[0010] Clean the obtained raw data, including deduplication, formatting, outlier detection and missing value filling;

[0011] Format the cleaned data according to the preset standardization rules, including encoding format, field mapping and data type conversion;

[0012] Encrypt the standardized data and set data access permissions to ensure data security and privacy;

[0013] Fuse the standardized multi-source data to eliminate redundant information and ensure data consistency and integrity;

[0014] Store the fused data into a target database and support dynamic expansion and data version management;

[0015] The standardized data service is provided to external applications through an API interface, and real-time query and analysis of the data are supported.

[0016] In an optional embodiment, the step of obtaining raw data from multiple data sources, including but not limited to structured data sources, semi-structured data sources and unstructured data sources, includes identifying and classifying data sources, connecting to structured data sources through JDBC or ODBC drivers to connect to relational databases, obtaining semi-structured data using built-in JSON libraries in programming languages to parse JSON files, obtaining JSON format data from other systems through API requests, collecting unstructured data using text processing tools to extract and clean text data, converting text data into structured or semi-structured format for subsequent analysis, and using programming languages to write scripts to periodically obtain data from different data sources, and using task scheduling tools to schedule the execution time of data acquisition tasks.

[0017] In an optional embodiment, the step of cleaning the obtained raw data includes identifying duplicate data by checking the unique identifier or field combination of the data records, using tools or programming languages to delete duplicate data records to ensure the uniqueness of each data, converting string data to a uniform case form, removing unnecessary spaces or special characters, identifying data points that deviate significantly from the normal range through statistical methods or business rules, marking outliers in the data for processing in subsequent analysis, checking whether there are missing values in the data set, using the average value of the field to fill in the missing values, using machine learning models to predict missing values, and deleting records containing missing values when there are too many missing values or cannot be reasonably filled.

[0018] In an optional embodiment, the step of encrypting the standardized data and setting data access permissions to ensure data security and privacy includes encrypting sensitive fields, encrypting the entire file, encrypting sensitive data in the database, using an encryption library to encrypt sensitive fields, using the built-in encryption function of the database to ensure the security of the encryption key, avoiding key leakage, using a secure random number to generate a key, storing the key in a secure location, regularly changing the key to reduce the risk of key leakage, setting data access permissions, and permission management is an important part of ensuring data security, controlling data access permissions, using multi-factor authentication to ensure the authenticity of user identity, and setting different data access permissions for different roles based on role-based access control.

[0019] In an alternative embodiment, the step of fusing the standardized multi-source data, eliminating redundant information, ensuring data consistency and integrity includes mapping fields in different data sources to a unified data model, ensuring consistent field meanings, unifying data coding standards to avoid data errors caused by inconsistent coding, matching the same entities from different data sources through unique identifiers or similarity algorithms, merging the matched data, eliminating redundant information, retaining the latest or most accurate data, ensuring consistency of the fused data between different fields, avoiding contradictory data, ensuring that all necessary data fields are included without omission, encrypting sensitive data to prevent data leakage, using a distributed database to store the fused data to improve data scalability and availability, regularly backing up data to ensure data security and recoverability.

[0020] In an alternative embodiment, the data fusion includes feature matching and fusion decision based on the constructed feature vector, feature matching is performed using a graph matching-based method, and fusion decision is designed based on the confidence of different data to determine the fusion weight of each point; an adjacency graph of feature points is constructed, and corresponding feature points between different source data are found through graph topology matching;

[0021] The fusion decision rule is that for matched points, the fusion weight is proportional to the confidence, and the expression formula is:

[0022]

[0023] Wherein, confidence i is the confidence of the point in the i th source point cloud data, ω i is the fusion weight of the point in the i th source point data.

[0024] In the weighted fusion calculation, the matched points are processed using the following weighted fusion calculation method according to the fusion weight, the coordinates of the corresponding points of each data source are multiplied by the fusion weight, and the results of all data sources are accumulated to obtain the coordinates of the fused point cloud data;

[0025] The processing process is: let the coordinates of a matched point in the i th data source fusion data be (x i ,y i ,z i ), the corresponding fusion weight be ω i , and n is the number of data sources, and the coordinates (x f ,y f ,z f ) of the point in the fused data are calculated by the following formula: and

[0026] In one optional implementation, the steps of providing standardized data services to external applications via API interfaces to support real-time data querying and analysis include: defining API endpoints, supported HTTP methods, request parameters, and response formats; designing API functionality to support real-time querying and complex analysis operations; integrating authentication mechanisms, such as API keys, OAuth 2.0, or JWT, to ensure authorized users and applications can access data; implementing encrypted data transmission; using HTTPS to protect data security during transmission; using high-performance servers or load balancing technologies to handle high-concurrency requests and improve API response speed; introducing caching mechanisms, such as Redis, to cache infrequently changing data, reducing database load and accelerating response time; writing detailed API documentation; using tools such as Swagger or OpenAPI specifications to help developers understand how to use the API; developing testing tools and automated test scripts to verify API functionality and performance, ensuring its stability and reliability; deploying monitoring tools to monitor API performance, availability, and security in real time, promptly identifying and resolving potential problems; establishing maintenance plans; and regularly updating and optimizing the API to adapt to changes in business needs and technological advancements.

[0027] Another aspect of the present invention provides a transformation system based on multi-source data, comprising:

[0028] The data acquisition module is used to acquire raw data from multiple data sources;

[0029] The data cleaning module is used to clean and process the acquired raw data.

[0030] The data standardization module is used to standardize the format of the cleaned data according to preset standardization rules;

[0031] The data security and privacy protection module is used to encrypt standardized data and set permissions.

[0032] The data fusion module is used to fuse standardized multi-source data and eliminate redundant information;

[0033] The data storage and management module is used to store the merged data into the target database and supports dynamic expansion and data version management;

[0034] The data service module is used to provide standardized data services to external applications through API interfaces.

[0035] In another aspect, the present invention provides an electronic device comprising:

[0036] At least one memory stores computer-executable instructions non-transiently;

[0037] At least one processor, configured to run the computer-executable instructions,

[0038] The computer-executable instructions are executed by the processor to implement the above-described method for transforming multi-source data.

[0039] In another aspect, the present invention provides a computer-readable storage medium storing computer-executable instructions that, when executed by at least one processor, implement the multi-source data conversion method described above.

[0040] This invention acquires raw data from multiple data sources, including but not limited to structured, semi-structured, and unstructured data sources; cleans the acquired raw data, including deduplication, formatting, outlier detection, and missing value imputation; standardizes the cleaned data according to preset standardization rules, including encoding format, field mapping, and data type conversion; encrypts the standardized data and sets data access permissions to ensure data security and privacy; merges the standardized multi-source data to eliminate redundant information and ensure data consistency and integrity; stores the merged data in a target database, supporting dynamic expansion and data version management; and provides standardized data services to external applications through an API interface, supporting real-time data querying and analysis. This invention effectively protects data security through data encryption. Attached Figure Description

[0041] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0042] Figure 1 This is a flowchart provided in Embodiment 1 of the present invention;

[0043] Figure 2 This is a framework diagram of the system provided in Embodiment 4 of the present invention;

[0044] Figure 3 This is a block diagram of the electronic device provided in Embodiment 5 of the present invention;

[0045] Figure 4 This is a block diagram of a computer-readable storage medium provided in Embodiment 5 of the present invention. Detailed Implementation

[0046] The technical solutions of the present invention will now be described with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0047] Hereinafter, the terms "first," "second," etc., are used for descriptive convenience only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0048] In this invention, unless otherwise explicitly specified and limited, the term "connection" should be interpreted broadly. For example, "connection" can be a fixed mechanical connection, a detachable mechanical connection, or an integral part; or, "connection" can be a direct connection or an indirect connection through an intermediate medium. Furthermore, unless otherwise explicitly specified and limited, the term "coupling" should be interpreted broadly. For example, "coupling" can be a direct electrical connection, such as physical contact and electrical conduction between two components; it can also be understood as an electrical connection between different components in a circuit structure through physical lines capable of transmitting electrical signals, such as copper foil or wires on a printed circuit board (PCB), to transmit electrical signals; or, "coupling" can be an indirect electrical connection between two components through an intermediate medium; or, "coupling" can be an electrical connection between two components in a non-contact manner, such as an electrical connection between two components using capacitive coupling to transmit electrical signals.

[0049] In this embodiment of the invention, directional terms such as "up," "down," "left," and "right" may be defined relative to the orientation of the components shown in the accompanying drawings. It should be understood that these directional terms can be relative concepts, used for relative description and clarification, and can change accordingly depending on the orientation of the components in the accompanying drawings.

[0050] Example 1:

[0051] like Figure 1 As shown, this embodiment of the invention provides a transformation method based on multi-source data, comprising the following steps:

[0052] Step S100: Obtain raw data from multiple data sources, including but not limited to structured data sources, semi-structured data sources, and unstructured data sources;

[0053] Step S200: Clean the acquired raw data, including deduplication, formatting, outlier detection, and missing value imputation;

[0054] Step S300: Standardize the format of the cleaned data according to preset standardization rules, including encoding format, field mapping and data type conversion;

[0055] Step S400: Encrypt the standardized data and set data access permissions to ensure data security and privacy;

[0056] Step S500: Merge the standardized multi-source data to eliminate redundant information and ensure data consistency and integrity;

[0057] Step S600: Store the merged data in the target database, and support dynamic expansion and data version management;

[0058] Step S700: Provide standardized data services to external applications through API interfaces, supporting real-time data query and analysis.

[0059] In the above embodiments, the original data is deduplicated to remove duplicate data; the data is formatted to unify the data representation; outliers are detected and processed, including deleting outliers or performing interpolation; missing values ​​are filled using interpolation or context-based filling methods. Data standardization rules are defined, including encoding formats, field mappings, and data type conversion rules; the cleaned data is converted according to the standardization rules to generate data in a unified format; the converted data is verified to ensure it meets preset standardization requirements. The standardized data is encrypted using symmetric or asymmetric encryption algorithms; data access permissions are set, including user permissions and role permissions, to ensure data is only accessible to authorized users; sensitive data is anonymized using data desensitization techniques to protect user privacy. Semantic analysis is performed on multi-source data to eliminate data redundancy; based on preset fusion rules, multi-source data is integrated to generate a unified data view; the fused data undergoes consistency verification to ensure accuracy and integrity. The merged data is stored in the target database, supporting both relational and non-relational databases; it provides data version management, supporting historical data review and version control; it supports dynamic expansion, automatically adjusting storage resources according to changes in data scale. Standardized data services are provided to external applications through API interfaces; real-time data querying and analysis are supported, providing efficient response capabilities; and data visualization functions are provided, allowing users to view data in chart form.

[0060] Example 2:

[0061] like Figure 2As shown, based on Embodiment 1, the data sources mentioned in step S100 of this embodiment of the invention include, but are not limited to, structured data sources, semi-structured data sources, and unstructured data sources. The steps include identifying and classifying data sources, connecting to structured data sources via JDBC or ODBC drivers to connect to relational databases, acquiring semi-structured data by parsing JSON files using a built-in JSON library in a programming language, obtaining JSON-formatted data from other systems via API requests, collecting unstructured data by extracting and cleaning text data using text processing tools, converting text data into structured or semi-structured formats for subsequent analysis, automating data acquisition by writing scripts in a programming language, periodically acquiring data from different data sources, and using task scheduling tools to schedule the execution time of data acquisition tasks.

[0062] The steps for cleaning the acquired raw data include: identifying duplicate data by checking the unique identifiers or field combinations of data records; deleting duplicate data records using tools or programming languages ​​to ensure the uniqueness of each data entry; converting string data to a uniform uppercase and lowercase form; removing redundant spaces or special characters; identifying data points that significantly deviate from the normal range using statistical methods or business rules; marking outliers in the data for handling in subsequent analysis; checking for missing values ​​in the dataset; filling missing values ​​with the average value of the field; predicting missing values ​​using machine learning models; and deleting records containing missing values ​​when there are too many missing values ​​or they cannot be reasonably filled.

[0063] The steps for encrypting standardized data and setting data access permissions to ensure data security and privacy include encrypting sensitive fields, encrypting stored files as a whole, encrypting sensitive data in the database, encrypting sensitive fields using an encryption library, using the database's built-in encryption function to ensure the security of encryption keys and prevent key leakage, generating keys using secure random numbers, storing keys in a secure location, changing keys regularly to reduce the risk of key leakage, and setting data access permissions. Access control is a crucial aspect of ensuring data security; controlling data access permissions involves using multi-factor authentication to ensure the authenticity of user identities and role-based access control to set different data access permissions for different roles.

[0064] In the above embodiments, data sources are identified and categorized as follows: Structured data sources: such as relational databases (e.g., MySQL, PostgreSQL) and spreadsheets (e.g., Excel); Semi-structured data sources: such as JSON files, XML files, and CSV files; Unstructured data sources: such as text files, images, audio, video, and social media content.

[0065] Use database connection tools to connect to a relational database via JDBC (Java Database Connectivity) or ODBC (Open Database Connectivity) drivers. Use the SQL query language to extract the required data. Process spreadsheets: Use tools such as Pandas (a Python library) to read and process Excel files, converting spreadsheet data into a structured format for further processing.

[0066] Reading JSON files: Parse JSON files using built-in JSON libraries in programming languages ​​such as Python and JavaScript. Retrieve JSON-formatted data from other systems via API requests. Parsing XML files: Read and process XML data using XML parsing libraries (such as ElementTree inPython). Retrieve XML-formatted data via APIs or the file system. Processing CSV files: Read and parse CSV files using CSV libraries. Handle delimiters and encoding issues in the data to ensure correct data reading.

[0067] Acquiring unstructured data and processing text files: Use text processing tools (such as regular expressions) to extract and clean text data. Convert text data into structured or semi-structured formats for subsequent analysis. Processing image and multimedia data: Use image processing libraries (such as OpenCV) to read and process image files. Use audio and video processing tools to extract and convert multimedia data. Scraping social media content: Use social media APIs (such as Twitter API, Facebook Graph API) to obtain user-generated content. Process the data formats returned by the APIs (such as JSON) and store them as semi-structured or structured data.

[0068] Automated Data Acquisition: Set up automated scripts using programming languages ​​(such as Python) to periodically acquire data from different data sources. Use task scheduling tools (such as cron and Airflow) to schedule the execution of data acquisition tasks. Utilize ETL tools: Deploy ETL (Extract, Transform, Load) tools (such as Apache NiFi and Talend) to automate the data extraction, transformation, and loading process. Handle Errors and Log Recording During Data Acquisition: Implement exception handling mechanisms to capture and record error information during data acquisition. Use a logging system (such as Log4j and Python logging) to record key events and errors during the data acquisition process. For large-volume data sources, adopt pagination or incremental acquisition strategies to avoid performance bottlenecks caused by loading large amounts of data at once. Utilize parallel processing techniques to improve the efficiency of data acquisition and processing.

[0069] Preliminary Data Cleaning and Preprocessing: Deduplication and Formatting: The acquired data is deduplicated to remove duplicate records. Data format and encoding are standardized to ensure consistency. Handling Missing and Outlier Values: Missing values ​​are filled or marked to ensure data integrity. Outliers are detected and handled to prevent them from affecting subsequent analysis. Data Transformation: Unstructured data is converted to structured or semi-structured formats for easier processing and analysis. Necessary field mapping and type conversion are performed to ensure data meets the requirements of subsequent processing. Data Storage and Management: Selecting an Appropriate Storage System: Based on data type and scale, an appropriate storage system is selected, such as relational databases, NoSQL databases, or data lakes. Implementing Data Version Management: Version control is implemented on the acquired data, recording the data change history and supporting data retrospection and comparison. Ensuring Data Security and Privacy: Sensitive data is encrypted to ensure security during storage and transmission. Access permissions are set to control the scope of data access and prevent unauthorized access.

[0070] Verify the accuracy of data acquisition: Data quality check: Perform quality checks on the acquired data to ensure its accuracy and completeness. Use data validation tools or scripts to check the data's format, range, and consistency. Sample data review: Manually review sample data to confirm the correctness and reliability of the data acquisition process. Automated testing: Write automated test cases to verify key steps and results in the data acquisition process, ensuring the stability and reliability of the data acquisition. Data acquisition documentation: Record detailed information about the data source, data acquisition methods and tools, data cleaning and preprocessing steps, etc., providing clear guidance and reference for subsequent data processing and analysis. Regular maintenance and updates: Regularly check the availability and stability of the data source and update the data acquisition methods and tools. Adjust the scope and strategy of data acquisition according to changes in business needs to ensure the continued effectiveness and applicability of data acquisition. Convert fields from one data type to another, such as converting string dates to date types and character numeric values ​​to numeric types. Field splitting and merging: Split or merge fields as needed, such as splitting an address field into subfields such as province, city, and district.

[0071] Data Validation and Quality Inspection: Data Integrity Check: Ensure all necessary fields have data and no omissions. Data Consistency Check: Ensure logical consistency between different fields, such as matching age and date of birth. Data Accuracy Check: Ensure data accuracy through sampling or comparison with the original data source. Automated Testing: Develop automated test scripts to verify the quality of the cleaned data, ensuring the stability and reliability of the cleaning process. Data Storage and Management: Store cleaned data in the target database or file, ensuring data security and accessibility. Implement Data Version Management: Record different versions of the cleaned data, supporting data backtracking and comparison. Dynamic Expansion and Maintenance: Dynamically adjust storage resources according to changes in data scale, ensuring efficient and scalable data processing.

[0072] Encryption Scope: Field-level encryption: Encrypt sensitive fields (such as passwords, ID numbers, mobile phone numbers, etc.). File-level encryption: Encrypt stored files (such as CSV, Excel, etc.). Database encryption: Encrypt sensitive data in the database. Symmetric encryption: Fast, suitable for encrypting large amounts of data (e.g., AES-256). Asymmetric encryption: Suitable for data transmission encryption (e.g., RSA). Hash encryption: Suitable for password storage (e.g., SHA-256, bcrypt).

[0073] Implement encryption for sensitive fields: Use encryption libraries (such as Python's cryptography library) to encrypt sensitive fields. For example, use hash encryption for password fields and store the hash value instead of the plaintext. File encryption: Use encryption tools (such as GPG or OpenSSL) to encrypt files. Generate encryption keys and ensure their secure storage. Database encryption: Use the database's built-in encryption features (such as Oracle's transparent data encryption or MySQL's table encryption). Ensure the security of encryption keys and prevent key leakage.

[0074] Encryption Key Management: Key Generation: Generate keys using secure random numbers. Key Storage: Store keys in a secure location (e.g., Hardware Security Module (HSM), Key Management Service (KMS)). Key Rotation: Regularly rotate keys to reduce the risk of key leakage. Data Access Permission Settings: Access control is a crucial aspect of ensuring data security, requiring strict control over data access permissions. User Authentication: Authentication using Multi-Factor Authentication (MFA) to ensure the authenticity of user identities. User Role Management: Define roles based on user responsibilities (e.g., administrator, data analyst, regular user).

[0075] Role-Based Access Control (RBAC): Set different data access permissions for different roles. For example, administrators can view and modify all data, while analysts can only view specific data. Ensure users only have the minimum permissions required to complete their tasks. For example, ordinary users can only view some data and cannot modify or delete it. Data Classification: Classify data according to its sensitivity (e.g., public, internal, confidential). Data Tagging: Add tags to data (e.g., "sensitive," "privacy") to facilitate access management and auditing. Access Logs: Record all data access operations, including user, time, and operation type. Real-Time Monitoring: Use security tools (e.g., SIEM) to monitor data access behavior in real time and detect abnormal activity. Regular Auditing: Regularly review data access logs to ensure that permission settings comply with security policies.

[0076] Data transmission security requires ensuring data security during transmission to prevent eavesdropping or tampering. This is achieved through several methods: HTTPS (using HTTPS protocol): Ensuring data encryption during transmission. SSL / TLS: Ensuring data transmission is encrypted using the SSL / TLS protocol in database connections. Digital signatures for data integrity verification: Digitally signing transmitted data ensures its integrity and authenticity. Checksums: Using checksums (such as CRC and MD5) to verify that data has not been tampered with during transmission. Dynamic encryption: Dynamically encrypting data transmitted in real-time ensures that data remains encrypted throughout the transmission process.

[0077] Data storage security requires further enhancement during the data storage phase. This includes: Database firewall: Deploying a database firewall to prevent unauthorized access. Access control: Restricting database access permissions to allow only authorized users. Data backup: Regularly backing up data and storing backups in secure locations (such as cloud storage or offline storage). Data recovery: Ensuring backup data can be quickly restored to prevent data loss. Data anonymization: Performing anonymization processing (such as replacement or masking) on ​​sensitive data to ensure its security when used in non-production environments.

[0078] Develop security strategies and data security policies: Develop detailed data security policies that clearly define requirements for data encryption, access permissions, transmission, and storage. Compliance: Ensure data processing complies with relevant laws and regulations (such as GDPR and CCPA). Employee Training: Conduct regular data security training for employees to improve their security awareness. Emergency Drills: Simulate data breaches to enhance the team's emergency response capabilities. Vulnerability Scanning: Regularly scan data storage and transmission systems for vulnerabilities to identify potential risks. Penetration Testing: Verify the effectiveness of data security measures through penetration testing. Updating Encryption Algorithms: Update encryption algorithms promptly based on technological advancements and changes in security threats. Optimizing Access Control Settings: Dynamically adjust data access permissions according to changes in business needs.

[0079] By encrypting standardized data and setting strict data access permissions, data security and privacy can be effectively guaranteed. Specific steps include data encryption, access control, secure transmission, secure storage, security policy formulation, and regular checks and updates. These measures minimize the risk of data leakage and tampering, ensuring data integrity and availability.

[0080] Example 3:

[0081] like Figure 1 As shown in Example 2, the steps provided in this embodiment of the invention for fusing standardized multi-source data, eliminating redundant information, and ensuring data consistency and integrity include mapping fields from different data sources to a unified data model to ensure consistent field meanings, unifying data encoding standards to avoid data errors caused by inconsistent encoding, matching identical entities from different data sources using unique identifiers or similarity algorithms, merging the matched data to eliminate redundant information, retaining the latest or most accurate data, ensuring consistency of the fused data across different fields to avoid contradictory data, ensuring that all necessary data fields are included without omission, encrypting sensitive data to prevent data leakage, using a distributed database to store the fused data to improve data scalability and availability, and regularly backing up data to ensure data security and recoverability.

[0082] The data fusion includes feature matching and fusion decision. Based on the constructed feature vector, a graph matching method is used for feature matching, and a fusion decision is designed according to the confidence of different data to determine the fusion weight of each point. An adjacency graph of feature points is constructed, and corresponding feature points between different source data are found through graph topology matching.

[0083] The fusion decision rule is as follows: For a matching point, the fusion weight is proportional to the confidence level, expressed by the following formula:

[0084]

[0085] Among them, confidence i Let ω be the confidence score of a point in the i-th source point cloud data. i Let be the fusion weight of the points in the i-th source data;

[0086] In the weighted fusion calculation, the matching points are processed according to the fusion weights using the following weighted fusion calculation method: the coordinates of the corresponding points of each data source are multiplied by the fusion weights, and the results of all data sources are summed to obtain the coordinates of the fused point cloud data.

[0087] The processing procedure is as follows: Let the coordinates of a matching point in the fused data of the i-th data source be (x... i ,y i ,z i The corresponding fusion weight is ω. i ,and n represents the number of data sources, and the coordinates (x, y) of that point in the merged data. f ,y f ,z f ) Calculated using the following formula: and

[0088] The steps for providing standardized data services to external applications via API interfaces, supporting real-time data querying and analysis, include: defining API endpoints, supported HTTP methods, request parameters, and response formats; designing API functionality to support real-time querying and complex analysis operations; integrating authentication mechanisms, such as API keys, OAuth 2.0, or JWT, to ensure authorized users and applications can access data; implementing encrypted data transmission; using HTTPS to protect data security during transmission; using high-performance servers or load balancing technologies to handle high-concurrency requests and improve API response speed; introducing caching mechanisms, such as Redis, to cache infrequently changing data, reducing database load and accelerating response time; writing detailed API documentation; using tools such as Swagger or OpenAPI specifications to help developers understand how to use the API; developing testing tools and automated test scripts to verify API functionality and performance, ensuring its stability and reliability; deploying monitoring tools to monitor API performance, availability, and security in real time, promptly identifying and resolving potential problems; establishing a maintenance plan; and regularly updating and optimizing the API to adapt to changes in business needs and technological advancements.

[0089] Example 4

[0090] Another aspect of the present invention provides a transformation system based on multi-source data, comprising:

[0091] The data acquisition module is used to acquire raw data from multiple data sources;

[0092] The data cleaning module is used to clean and process the acquired raw data.

[0093] The data standardization module is used to standardize the format of the cleaned data according to preset standardization rules;

[0094] The data security and privacy protection module is used to encrypt standardized data and set permissions.

[0095] The data fusion module is used to fuse standardized multi-source data and eliminate redundant information;

[0096] The data storage and management module is used to store the merged data into the target database and supports dynamic expansion and data version management;

[0097] The data service module is used to provide standardized data services to external applications through API interfaces.

[0098] In the above embodiments, the data cleaning module includes the following sub-modules: a data deduplication sub-module for removing duplicate data; a data formatting sub-module for standardizing data representation; an outlier detection and processing sub-module for detecting and processing outliers; and a missing value filling sub-module for filling missing values. The data standardization module includes the following sub-modules: a standardization rule definition sub-module for defining data standardization rules; a data format conversion sub-module for converting the cleaned data according to the standardization rules; and a data verification sub-module for verifying whether the converted data meets the preset standardization requirements. The data security and privacy protection module includes the following sub-modules: a data encryption sub-module for encrypting the standardized data; a data access permission setting sub-module for setting data access permissions; and a data anonymization sub-module for anonymizing sensitive data. The data fusion module includes the following sub-modules: a data redundancy elimination sub-module for eliminating data redundancy; a data integration sub-module for integrating multi-source data based on preset fusion rules; and a data consistency verification sub-module for verifying whether the fused data meets consistency requirements. The data storage and management module includes the following sub-modules: Data storage sub-module: used to store the merged data into the target database; Data version management sub-module: used to support historical data backtracking and version control; Dynamic expansion sub-module: used to automatically adjust storage resources according to changes in data scale. The data service module includes the following sub-modules: Data service interface sub-module: used to provide standardized data services through API interfaces; Data query and analysis sub-module: used to support real-time data query and analysis; Data visualization sub-module: used to provide data visualization display functions.

[0099] Example 5:

[0100] Figure 3 A block diagram of an exemplary electronic device suitable for implementing embodiments of the present invention is shown.

[0101] The electronic device may include a central processing unit / microprocessor / main control chip, etc. 4; and a storage medium 5, coupled to the central processing unit / microprocessor / main control chip, etc. 4, and storing computer-executable instructions therein for performing the steps of various methods of embodiments of the present invention when executed by the processor.

[0102] The central processing unit / microprocessor / main control chip, etc., can include, but are not limited to, one or more processors or microprocessors.

[0103] Storage medium 5 may include, but is not limited to, random access memory (RAM), read-only memory (ROM), flash memory, EPROM memory, EEPROM memory, registers, computer storage media (e.g., hard disk, floppy disk, solid-state drive, removable disk, CD-ROM, DVD-ROM, Blu-ray disc, etc.).

[0104] In addition, the electronic device may also include (but is not limited to) a data bus 6, an input / output bus / external bus / device bus 7, a display 8, and input / output devices 9 (e.g., keyboard, mouse, speaker, etc.).

[0105] The central processing unit / microprocessor / main control chip, etc. 4 can communicate with external devices (8, 9, etc.) via I / O bus 7 through wired or wireless network (not shown).

[0106] The storage medium 5 may also store at least one computer-executable instruction for performing the steps of various functions and / or methods in the embodiments described herein when the central processing unit / microprocessor / main control chip, etc., 4 is running.

[0107] In one embodiment, the at least one computer-executable instruction may also be compiled into or comprise a software product, wherein one or more computer-executable instructions are executed by a processor to perform the steps of the various functions and / or methods in the embodiments described herein.

[0108] Figure 4 A schematic diagram of a computer-readable storage medium according to an embodiment of the present invention is shown.

[0109] like Figure 4As shown, the non-transitory computer-readable storage medium 11 stores instructions, such as computer-readable instructions 10. When the computer-readable instructions 10 are executed by a processor, the various methods described above can be performed. The non-transitory computer-readable storage medium includes, but is not limited to, volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-transitory non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. For example, the non-transitory computer-readable storage medium 11 can be connected to a computing device such as a computer, and then, when the computing device executes the computer-readable instructions 10 stored on the computer-readable storage medium 11, the various methods described above can be performed.

[0110] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0111] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0112] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0113] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for executing all or part of the steps of the methods of the various embodiments of this invention through a computer device (which may be a personal computer, server, or network device, etc.). The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0114] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A transformation method based on multi-source data, characterized in that, Includes the following steps: Raw data is obtained from multiple data sources, including but not limited to structured data sources, semi-structured data sources, and unstructured data sources; The acquired raw data is cleaned, including deduplication, formatting, outlier detection, and missing value imputation. The cleaned data is formatted according to preset standardization rules, which include encoding format, field mapping, and data type conversion. The standardized data is encrypted, and data access permissions are set to ensure data security and privacy. Standardized multi-source data are merged to eliminate redundant information and ensure data consistency and integrity. The merged data is stored in the target database, and supports dynamic expansion and data version management; Standardized data services are provided to external applications through API interfaces, supporting real-time data querying and analysis.

2. The transformation method based on multi-source data as described in claim 1, characterized in that, The steps of acquiring raw data from multiple data sources, including but not limited to structured data sources, semi-structured data sources, and unstructured data sources, include: identifying and classifying data sources; connecting to structured data sources by connecting to relational databases via JDBC or ODBC drivers; acquiring semi-structured data by parsing JSON files using the built-in JSON library of a programming language; obtaining JSON-formatted data from other systems via API requests; collecting unstructured data by extracting and cleaning text data using text processing tools; converting text data into structured or semi-structured formats for subsequent analysis; automating data acquisition by writing scripts in a programming language; periodically acquiring data from different data sources; and scheduling the execution time of data acquisition tasks using task scheduling tools.

3. The transformation method based on multi-source data as described in claim 1, characterized in that, The steps for cleaning the acquired raw data include: identifying duplicate data by checking the unique identifiers or field combinations of data records; deleting duplicate data records using tools or programming languages ​​to ensure the uniqueness of each data entry; converting string data to a uniform uppercase and lowercase form; removing redundant spaces or special characters; identifying data points that significantly deviate from the normal range using statistical methods or business rules; marking outliers in the data for handling in subsequent analysis; checking for missing values ​​in the dataset; filling missing values ​​with the average value of the field; predicting missing values ​​using machine learning models; and deleting records containing missing values ​​when there are too many missing values ​​or they cannot be reasonably filled.

4. The transformation method based on multi-source data as described in claim 1, characterized in that, The steps for encrypting standardized data and setting data access permissions to ensure data security and privacy include encrypting sensitive fields, encrypting stored files as a whole, encrypting sensitive data in the database, encrypting sensitive fields using an encryption library, using the database's built-in encryption function to ensure the security of encryption keys and prevent key leakage, generating keys using secure random numbers, storing keys in a secure location, changing keys regularly to reduce the risk of key leakage, and setting data access permissions. Access control is a crucial aspect of ensuring data security; controlling data access permissions involves using multi-factor authentication to ensure the authenticity of user identities and role-based access control to set different data access permissions for different roles.

5. The transformation method based on multi-source data as described in claim 1, characterized in that, The steps for fusing standardized multi-source data, eliminating redundant information, and ensuring data consistency and integrity include mapping fields from different data sources to a unified data model to ensure consistent field meanings, unifying data encoding standards to avoid data errors caused by inconsistent encoding, matching identical entities from different data sources using unique identifiers or similarity algorithms, merging the matched data to eliminate redundant information, retaining the latest or most accurate data, ensuring consistency of the fused data across different fields to avoid contradictory data, ensuring all necessary data fields are included without omission, encrypting sensitive data to prevent data leakage, using a distributed database to store the fused data to improve data scalability and availability, and regularly backing up data to ensure data security and recoverability.

6. The transformation method based on multi-source data as described in claim 1, characterized in that, The data fusion includes feature matching and fusion decision. Based on the constructed feature vector, a graph matching method is used for feature matching, and a fusion decision is designed according to the confidence of different data to determine the fusion weight of each point. An adjacency graph of feature points is constructed, and corresponding feature points between different source data are found through graph topology matching. The fusion decision rule is as follows: For a matching point, the fusion weight is proportional to the confidence level, expressed by the following formula: Among them, confidence i Let ω be the confidence score of a point in the i-th source point cloud data. i Let be the fusion weight of the points in the i-th source data; In the weighted fusion calculation, the matching points are processed according to the fusion weights using the following weighted fusion calculation method: the coordinates of the corresponding points of each data source are multiplied by the fusion weights, and the results of all data sources are summed to obtain the coordinates of the fused point cloud data. The processing procedure is as follows: Let the coordinates of a matching point in the fused data of the i-th data source be (x... i ,y i ,z i The corresponding fusion weight is ω. i ,and n represents the number of data sources, and the coordinates (x, y) of that point in the merged data. f ,y f ,z f ) Calculated using the following formula: and 7. The transformation method based on multi-source data as described in claim 1, characterized in that, The steps for providing standardized data services to external applications via API interfaces, supporting real-time data querying and analysis, include: defining API endpoints, supported HTTP methods, request parameters, and response formats; designing API functionality to support real-time querying and complex analysis operations; integrating authentication mechanisms, such as API keys, OAuth 2.0, or JWT, to ensure authorized users and applications can access data; implementing encrypted data transmission; using HTTPS to protect data security during transmission; using high-performance servers or load balancing technologies to handle high-concurrency requests and improve API response speed; introducing caching mechanisms, such as Redis, to cache infrequently changing data, reducing database load and accelerating response time; writing detailed API documentation; using tools such as Swagger or OpenAPI specifications to help developers understand how to use the API; developing testing tools and automated test scripts to verify API functionality and performance, ensuring its stability and reliability; deploying monitoring tools to monitor API performance, availability, and security in real time, promptly identifying and resolving potential problems; establishing a maintenance plan; and regularly updating and optimizing the API to adapt to changes in business needs and technological advancements.

8. A multi-source data conversion system according to any one of claims 1 to 7, characterized in that, include: The data acquisition module is used to acquire raw data from multiple data sources; The data cleaning module is used to clean and process the acquired raw data. The data standardization module is used to standardize the format of the cleaned data according to preset standardization rules; The data security and privacy protection module is used to encrypt standardized data and set permissions. The data fusion module is used to fuse standardized multi-source data and eliminate redundant information; The data storage and management module is used to store the merged data into the target database and supports dynamic expansion and data version management; The data service module is used to provide standardized data services to external applications through API interfaces.

9. An electronic device, comprising: At least one memory stores computer-executable instructions non-transiently; At least one processor, configured to run the computer-executable instructions, The computer-executable instructions are executed by the processor at runtime, and the method for transforming multi-source data according to any one of claims 1-7 is described.

10. A computer-readable storage medium, wherein, The computer-readable storage medium stores computer-executable instructions, which, when executed by at least one processor, implement a multi-source data conversion method according to any one of claims 1-7.

Images