Data processing method and system

Abstract

The invention relates to a data processing method and system, and belongs to the technical field of data processing.The method comprises the steps that resource data are obtained, and a resource file corresponding to the resource data is generated; based on an increment comparison rule, comparing the resource data of the resource file with the metadata in the twin library to obtain incremental data, and sending the incremental data and a label of the incremental data to a modular conversion system; based on a modular conversion rule, performing modular conversion processing on the incremental data in the modular conversion system, storing the modular-converted incremental data into a modular conversion library, and forming an incremental storage log; and sending the incremental warehousing log to a whole book library system. According to the method, high efficiency and accuracy of data processing are realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of data processing, and in particular, relates to a data processing method and system. Background Art

[0002] With the rapid development of information technology, the digitalization process of various industries is accelerating, and data has become the core asset of enterprise operations and decision-making. In many business scenarios, such as financial transaction processing, e-commerce platform operations, industrial production monitoring, etc., data presents the characteristics of massive, heterogeneous and continuously updated. Therefore, how to better realize data processing has become an urgent problem to be solved. Summary of the invention

[0003] In view of the above deficiencies in the prior art, an object of the present invention is to provide a data processing method and system.

[0004] The first aspect of the present invention proposes a data processing method, including: S1, obtaining resource data and generating a resource file corresponding to the resource data; S2, based on the incremental comparison rule, comparing the resource data of the resource file with the metadata in the twin library to obtain incremental data, and sending the incremental data and the label of the incremental data to the model-to-model system; S3, based on the model-to-model rule, performing model-to-model processing on the incremental data in the model-to-model system, storing the incremental data after model-to-model conversion into the model-to-model library, and forming an incremental storage log; S4, sending the incremental storage log to the whole library system.

[0005] Furthermore, based on the incremental comparison rules, the resource data of the resource file is compared with the metadata in the twin library to obtain incremental data, including: determining the data type of the metadata; parsing the resource data into target data corresponding to the data type; comparing the target data with the metadata to obtain the incremental data.

[0006] Further, determining the label of the incremental data includes: determining whether the target data does not exist in the metadata; when it is determined that the target data does not exist in the metadata, determining that the label of the target data is added; when it is determined that the target data exists in the metadata, determining whether the fields of the metadata and the target data are the same; when it is determined that the fields of the metadata and the target data are not the same, determining that the label of the target data is updated; when it is determined that the fields of the metadata and the target data are the same, determining that the label of the target data is maintained.

[0007] Furthermore, it also includes: judging whether the metadata does not exist in the target data; and if it is determined that the metadata does not exist in the target data, determining that the label of the target data is deleted.

[0008] Furthermore, based on the model conversion rules, the incremental data in the model conversion system is model converted, including: mapping the incremental data to a preset unified model, the preset unified model including target data naming and target data structure form; analyzing the data of the preset unified model to obtain data elements of the data; based on the data elements, dividing the preset unified model into entity model conversion, extended entity model conversion, relationship model conversion and extended relationship model conversion.

[0009] Furthermore, the data elements include entities, extended entities, relationships and extended relationships.

[0010] Furthermore, it also includes: manually identifying the incremental storage log, and when it is identified that the incremental storage log is abnormal, manually correcting the abnormal data.

[0011] The second aspect of the present invention proposes a data processing system, including: an acquisition module, used to acquire resource data and generate a resource file corresponding to the resource data; a comparison module, used to compare the resource data of the resource file with the metadata in the twin library based on incremental comparison rules to obtain incremental data, and send the incremental data and the label of the incremental data to the model-to-model system; a model-to-model module, used to perform model-to-model processing on the incremental data in the model-to-model system based on model-to-model rules, and store the incremental data after model-to-model conversion into the model-to-model library, and form an incremental storage log; a sending module, used to send the incremental storage log to the whole library system.

[0012] According to a third aspect of the present invention, an electronic device is proposed, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any of the methods described in the first aspect of the present invention.

[0013] According to a fourth aspect of the present invention, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any one of the methods according to the first aspect of the present invention.

[0014] The beneficial effects of the present invention are as follows:

[0015] The data processing method and system described in the present invention obtain resource data and generate a resource file corresponding to the resource data; based on the incremental comparison rule, the resource data of the resource file is compared with the metadata in the twin library to obtain incremental data, and the incremental data and the label of the incremental data are sent to the model-to-model system; based on the model-to-model rule, the incremental data in the model-to-model system is model-to-model processed, and the incremental data after model-to-model is stored in the model-to-model library, and an incremental storage log is formed; the incremental storage log is sent to the whole library system. This method achieves high efficiency and accuracy in data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings are only used to illustrate specific embodiments and are not considered to limit the present invention. In the entire drawings, the same reference symbols represent the same components. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings.

[0017] Figure 1 is a flow chart of a data processing method according to an embodiment of the present invention;

[0018] Figure 2 is a flow chart of a data processing method according to a specific embodiment of the present invention;

[0019] Figure 3 is a schematic diagram of a data processing system according to an embodiment of the present invention;

[0020] Figure 4 It is a structural block diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0021] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. It should be understood that these descriptions are only exemplary and are not intended to limit the scope of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in the field without making creative work should fall within the scope of protection of the present invention.

[0022] Furthermore, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessarily obscuring the concepts disclosed in the present invention.

[0023] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the orientation or position relationship indicated by the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inside", "outside", etc. is based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation of the present invention. In addition, the terms "first", "second", and "third" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. The terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be a connection between the two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0024] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of methods and systems consistent with some aspects of the present invention as detailed in the appended claims.

[0025] With the rapid development of information technology, the digitalization process of various industries is accelerating, and data has become the core asset of enterprise operations and decision-making. In many business scenarios, such as financial transaction processing, e-commerce platform operations, industrial production monitoring, etc., data presents the characteristics of massive, heterogeneous and continuously updated.

[0026] On the one hand, data sources are extremely wide, including sensor collection, user interaction records, business system data output, etc. These data have significant differences in format, structure and semantics, forming a complex heterogeneous data source environment. For example, in the IoT scenario, different types of sensors may transmit data in different protocols and data formats, some in binary format, and some in text data in a specific format. Enterprises need an effective way to integrate these data to achieve unified management and utilization of data.

[0027] On the other hand, enterprises usually have built a relatively stable core data management system, such as the twin library mentioned in the article, which stores important metadata information and represents the benchmark architecture and key business logic of enterprise data. In the process of continuous data generation and updating, how to ensure that the newly acquired resource data can be accurately compared and integrated with the metadata in the twin library has become a key issue. If handled improperly, it may lead to problems such as data redundancy, inconsistency, and data association errors, which will seriously affect the normal development of the enterprise business and the accuracy of decision-making.

[0028] At the same time, in order to meet the specific requirements of different business links and analysis needs for data formats, data needs to be converted into a schema. For example, data may be suitable for fast transmission in its original state of collection, but when it is stored in a data warehouse for data analysis and mining, it needs to be converted into a format suitable for the data warehouse architecture, such as a star or snowflake schema. This has given rise to the need to process the data into a schema and store it in a standardized manner in a special schema library. In this process, in order to ensure the traceability and monitoring of data processing, recording incremental storage logs is an essential link.

[0029] In addition, there are often multiple data storage and processing systems within an enterprise, and effective data interaction and collaboration are required between them. Sending incremental storage logs to the full storage system helps to achieve orderly data flow and sharing in the entire enterprise data ecosystem, ensuring that each system can perform corresponding operations and decisions based on the latest data status, thereby improving the overall data processing efficiency and business responsiveness of the enterprise. Therefore, how to better achieve data processing has become an urgent problem to be solved.

[0030] To this end, the present invention proposes a data processing method, system and related equipment. Specifically, the data processing method, system and related equipment of an embodiment of the present invention are described below with reference to the accompanying drawings.

[0031] Figure 1 1 is a flow chart of a data processing method according to an embodiment of the present invention. It should be noted that the data processing method of the embodiment of the present invention can be applied to the data processing system of the embodiment of the present invention. The data processing system can be configured on an electronic device or in a server. The present application embodiment does not limit this.

[0032] like Figure 1 As shown, the data processing method comprises:

[0033] S110, acquiring resource data and generating a resource file corresponding to the resource data.

[0034] In an embodiment of the present invention, resource data can be acquired in real time.

[0035] S120, based on the incremental comparison rules, compare the resource data of the resource file with the metadata in the twin library to obtain incremental data, and send the incremental data and the label of the incremental data to the model-to-model system.

[0036] In an embodiment of the present invention, the data type of the metadata is determined; the resource data is parsed into target data corresponding to the data type; the target data is compared with the metadata to obtain incremental data. Then, the label of the incremental data can be determined, so that the incremental data and the label of the incremental data are sent to the model conversion system. The specific implementation method can be referred to the subsequent embodiments.

[0037] S130, based on the model conversion rules, the incremental data in the model conversion system is model converted, and the incremental data after the model conversion is stored in the model conversion library, and an incremental storage log is formed.

[0038] S140, sending the incremental storage log to the full storage system.

[0039] According to the data processing method of the embodiment of the present invention, resource data is obtained, and a resource file corresponding to the resource data is generated; based on the incremental comparison rule, the resource data of the resource file is compared with the metadata in the twin library to obtain incremental data, and the incremental data and the label of the incremental data are sent to the model-to-model system; based on the model-to-model rule, the incremental data in the model-to-model system is model-converted, and the incremental data after model-conversion is stored in the model-to-model library, and an incremental storage log is formed; the incremental storage log is sent to the whole library system. This method achieves high efficiency and accuracy in data processing.

[0040] In order to make it easier for those skilled in the art to understand the present invention, Figure 2 is a data processing method according to a specific embodiment of the present invention, such as Figure 2 As shown, the data processing method comprises:

[0041] S210, acquiring resource data and generating a resource file corresponding to the resource data.

[0042] S220, based on the incremental comparison rule, compare the resource data of the resource file with the metadata in the twin library to obtain incremental data.

[0043] In an embodiment of the present invention, the data type of the metadata is determined; the resource data is parsed into target data corresponding to the data type; the target data is compared with the metadata to obtain incremental data.

[0044] The data types include but are not limited to devices, ports, boards, etc.

[0045] S230, determining the label of the incremental data.

[0046] In an embodiment of the present invention, it is determined whether the target data does not exist in the metadata; when it is determined that the target data does not exist in the metadata, the label of the target data is determined to be added; when it is determined that the target data exists in the metadata, it is determined whether the fields of the metadata and the target data are the same; when it is determined that the fields of the metadata and the target data are not the same, the label of the target data is determined to be updated; when it is determined that the fields of the metadata and the target data are the same, the label of the target data is determined to be retained.

[0047] Among them, determining that the target data does not exist in the metadata can be understood as the existence of data in the resource file that is not in the twin library.

[0048] In an embodiment of the present invention, it is determined whether there is no metadata in the target data; if it is determined that there is no metadata in the target data, the label of the target data is determined to be deleted.

[0049] Among them, determining that there is no metadata in the target data can be understood as the resource file does not have the data in the twin library.

[0050] S240, sending the incremental data and the label of the incremental data to the model-to-model system.

[0051] S250, based on the model conversion rules, the incremental data in the model conversion system is model converted, and the incremental data after the model conversion is stored in the model conversion library, and an incremental storage log is formed.

[0052] In an embodiment of the present invention, incremental data is mapped to a preset unified model, which includes target data naming and target data structure form; the data of the preset unified model is analyzed to obtain data elements of the data; based on the data elements, the preset unified model is divided into entity model conversion, extended entity model conversion, relationship model conversion and extended relationship model conversion.

[0053] Among them, data elements include entities, extended entities, relationships and extended relationships.

[0054] S260, sending the incremental storage log to the full storage system.

[0055] In an embodiment of the present invention, the incremental storage log is manually identified, and when an abnormality is identified in the incremental storage log, the abnormal data is manually corrected.

[0056] In a specific embodiment of the present invention, the resource sharing platform manages the twin library metadata of the network element configuration through the meta platform, and the meta platform encapsulates the twin library service of the network element configuration; the models of the network element configuration model conversion library and the unified resource model database are uniformly managed through metadata tools to guide model upgrades and provide data services. Among them, 1) the acquisition and control platform completes real-time / quasi-real-time / scheduled data acquisition according to the conditions of the acquisition source, and provides an interface for the resource sharing platform to call to obtain resource data, and generate corresponding resource files; 2) users configure incremental comparison rules, for example: data updates when the device name changes, and data updates when the device belongs to a park; users configure model conversion rules, for example: the device name business attribute model is converted into the unified model device table NAME, and the device belongs to the park attribute model is converted into the park entity; 3) based on the incremental comparison rules, the resource data of the resource file is compared with the metadata in the twin library to obtain incremental data, wherein the data comparison includes: identifying the changes in the resource data of the resource file and the metadata in the twin library, and marking these incremental data and pushing them to the model conversion system, wherein, if the twin library does not have the resource file, The data is a newly added label; if the twin library has the data in the resource file and the data has changed, the label is updated; if the twin library has the data in the resource file and the data has not changed, the label is kept; if the twin library has data that the resource file does not have, the label is deleted; 4) Based on the simulation rules, the incremental data in the simulation system is simulated, that is, the resource data of different manufacturers are simulated by a preset unified model, so that the data of different manufacturers are normalized and stored according to the preset unified model specifications; the incremental data is simulated according to the unified model, and each type of data is simulated according to the assigned major and minor specifications and specifications, and the attribute range is simulated according to the fields defined by this specification. After processing, the meta-model data is simulated and spliced ​​from the network element configuration twin data model to the unified resource model, and the data is written to the network element configuration simulation database. The types of model conversion are divided into: entity model conversion (such as equipment, ports), extended entity model conversion (data without connection, such as the park, is not collected, and the park is modeled out according to the field of the park to which the equipment belongs), relationship model conversion (equipment to which the port belongs), and extended relationship model conversion (the relationship of the park to which the equipment belongs is not collected, and a new relationship needs to be added); 5) The incremental data after model conversion is stored in the model conversion library, and an incremental storage log is formed; 6) The incremental storage log is sent to the whole library system. It can be seen that the big data incremental comparison part effectively controls the changed data to be modeled and stored, and improves the performance of the data flow process; improves the configurable method of model conversion and the big data service data comparison and model conversion according to different acquisition and control plug-in business models and unified metadata model model configuration items, and the configuration model conversion part can make the data modeled according to the user-configured model conversion rules, support real-time modification of configuration and dynamic reading of configuration, so that the data collection model conversion is seamlessly connected, which greatly guarantees the flexibility of data in the model conversion process and improves the timeliness of the data flow throughout the process.

[0057] According to the data processing method of the embodiment of the present invention, by comparing the metadata in the resource file with the twin library, only the newly added, updated or deleted data (i.e., incremental data) is processed, thereby avoiding repeated processing of the full amount of data and significantly improving the data processing efficiency; the data type of the resource data is parsed to ensure accurate comparison with the target metadata, further improving the accuracy of data processing; through detailed label determination logic (new, updated, retained, deleted), the change of data can be clearly identified, providing strong support for subsequent data processing; the twin library metadata of the network element configuration is managed through the meta-platform, ensuring the consistency and accuracy of the data source; through the incremental comparison rules and model conversion rules, the seamless connection and conversion between the resource data and the twin library metadata is achieved, ensuring the data Consistency and integrity; the generation and storage of incremental warehousing logs provide strong support for data traceability and auditing, further enhancing the integrity and credibility of the data; users can configure incremental comparison rules and model conversion rules according to actual needs, making the data processing process more flexible and customizable; by presetting a unified model to convert data, the data of different manufacturers can be normalized and stored in accordance with unified specifications, enhancing the scalability and interoperability of data; it supports multiple model conversion types such as entity model conversion, extended entity model conversion, relationship model conversion and extended relationship model conversion, meeting the data processing needs in different scenarios; since only incremental data is processed, the repeated storage of the full data is avoided, saving storage space; by reducing the amount of data processing, the demand for computing resources is reduced, thereby saving costs.

[0058] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of each embodiment of the present application.

[0059] According to one aspect of an embodiment of the present invention, a data processing system is also provided. Figure 3 is a schematic diagram of a data processing system according to an embodiment of the present invention; Figure 3 As shown, including:

[0060] The acquisition module 310 is used to acquire resource data and generate a resource file corresponding to the resource data;

[0061] A comparison module 320 is used to compare the resource data of the resource file with the metadata in the twin library based on the incremental comparison rule to obtain incremental data, and send the incremental data and the label of the incremental data to the model-to-model system;

[0062] The model conversion module 330 is used to perform model conversion processing on the incremental data in the model conversion system based on model conversion rules, store the incremental data after model conversion into a model conversion library, and form an incremental storage log;

[0063] The sending module 340 is used to send the incremental storage log to the full storage system.

[0064] According to the data processing system of the embodiment of the present invention, resource data is obtained and a resource file corresponding to the resource data is generated; based on the incremental comparison rule, the resource data of the resource file is compared with the metadata in the twin library to obtain incremental data, and the incremental data and the label of the incremental data are sent to the model-to-model system; based on the model-to-model rule, the incremental data in the model-to-model system is model-to-model processed, and the incremental data after model-to-model is stored in the model-to-model library, and an incremental storage log is formed; the incremental storage log is sent to the whole library system. In this way, the efficiency and accuracy of data processing are achieved.

[0065] Optionally, the comparison module 320 is specifically used to determine the data type of the metadata; parse the resource data into target data corresponding to the data type; and compare the target data with the metadata to obtain the incremental data.

[0066] Optionally, the comparison module 320 is specifically used to determine whether the target data does not exist in the metadata; when it is determined that the target data does not exist in the metadata, determine that the label of the target data is added; when it is determined that the target data exists in the metadata, determine whether the fields of the metadata and the target data are the same; when it is determined that the fields of the metadata and the target data are not the same, determine that the label of the target data is updated; when it is determined that the fields of the metadata and the target data are the same, determine that the label of the target data is maintained.

[0067] Optionally, the comparison module 320 is specifically configured to determine whether the metadata does not exist in the target data; if it is determined that the metadata does not exist in the target data, determine that the label of the target data is deleted.

[0068] Optionally, the model conversion module 330 is specifically used to map the incremental data to a preset unified model, wherein the preset unified model includes target data naming and target data structure form; analyze the data of the preset unified model to obtain data elements of the data; and based on the data elements, divide the preset unified model into entity model conversion, extended entity model conversion, relationship model conversion and extended relationship model conversion.

[0069] Optionally, the data elements include entities, extended entities, relationships and extended relationships.

[0070] Optionally, a manual identification module is further included, which is used to manually identify the incremental storage log, and when it is identified that the incremental storage log is abnormal, the abnormal data is corrected manually.

[0071] According to one aspect of an embodiment of the present invention, an electronic device is provided.

[0072] Figure 4 Schematic diagram of the structure of an electronic device according to an embodiment of the present invention. Figure 4 As shown, the electronic device may include one or more ( Figure 4 Only one is shown in the figure) processor 102 (processor 102 may include but is not limited to a microprocessor (Microprocessor Unit, referred to as MPU) or a programmable logic device (Programmable logic device, referred to as PLD)) and a memory 104 for storing data. In an exemplary embodiment, the electronic device may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 4 The structure shown is for illustration only and does not limit the structure of the above terminal device. Figure 4 More or fewer components as shown, or with Figure 4 Equivalent functions or comparisons shown Figure 4 A different configuration with more features is shown.

[0073] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the data processing method in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the terminal device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0074] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the switching device. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0075] The present invention provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute a data processing method.

[0076] The applicant of the present invention has made a detailed explanation and description of the implementation examples of the present invention in conjunction with the drawings in the specification. However, those skilled in the art should understand that the above implementation examples are only preferred implementation schemes of the present invention, and the detailed description is only to help readers better understand the spirit of the present invention, but not to limit the scope of protection of the present invention. On the contrary, any improvements or modifications based on the inventive spirit of the present invention should fall within the scope of protection of the present invention.

[0077] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0078] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present invention. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

[0079] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of the present invention, rather than to limit them. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the above embodiments, or replace some of the technical features therein by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. Any changes or replacements that can be easily thought of by those skilled in the art within the technical scope disclosed in the present invention should be covered within the protection scope of the present invention.

Claims

1. A data processing method, characterized in that: include: S1, obtaining resource data and generating a resource file corresponding to the resource data; S2, based on the incremental comparison rule, comparing the resource data of the resource file with the metadata in the twin library to obtain incremental data, and sending the incremental data and the label of the incremental data to the model conversion system; S3, based on the model conversion rules, performing model conversion processing on the incremental data in the model conversion system, storing the incremental data after model conversion into a model conversion library, and forming an incremental storage log; S4, sending the incremental storage log to the full storage system.

2. The data processing method according to claim 1, characterized in that: Based on the incremental comparison rules, the resource data of the resource file is compared with the metadata in the twin library to obtain incremental data, including: determining a data type of the metadata; Parsing the resource data into target data corresponding to the data type; The target data is compared with the metadata to obtain the incremental data.

3. The data processing method according to claim 2, characterized in that: Determining a label of the incremental data includes: Determining whether the target data does not exist in the metadata; When it is determined that the target data does not exist in the metadata, determining that the label of the target data is newly added; In the case where it is determined that the target data exists in the metadata, determining whether the fields of the metadata and the target data are the same; In a case where it is determined that the fields of the metadata and the target data are not the same, determining that the label of the target data is updated; When it is determined that the fields of the metadata and the target data are the same, the tag of the target data is determined to be retained.

4. The data processing method according to claim 3, characterized in that: Also includes: Determining whether the metadata does not exist in the target data; When it is determined that the metadata does not exist in the target data, the label of the target data is determined to be deleted.

5. The data processing method according to claim 1, characterized in that: Based on the analog conversion rule, the incremental data in the analog conversion system is subjected to analog conversion processing, including: Mapping the incremental data to a preset unified model, wherein the preset unified model includes target data naming and target data structure form; Analyze the data of the preset unified model to obtain data elements of the data; Based on the data elements, the preset unified model is divided into entity model conversion, extended entity model conversion, relationship model conversion and extended relationship model conversion.

6. The data processing method according to claim 5, characterized in that: The data elements include entities, extended entities, relationships and extended relationships.

7. The data processing method according to claim 1, characterized in that: Also includes: The incremental storage log is manually identified, and when an abnormality is identified in the incremental storage log, the abnormal data is manually corrected.

8. A data processing system, characterized in that: include: An acquisition module, used to acquire resource data and generate a resource file corresponding to the resource data; A comparison module, used for comparing the resource data of the resource file with the metadata in the twin library based on the incremental comparison rule to obtain incremental data, and sending the incremental data and the label of the incremental data to the model-to-model system; A model conversion module, used for performing model conversion processing on the incremental data in the model conversion system based on model conversion rules, storing the incremental data after model conversion into a model conversion library, and forming an incremental storage log; The sending module is used to send the incremental storage log to the full storage system.

9. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 7.

Images