Data processing method and device and electronic equipment

Abstract

The invention provides a data processing method and device and electronic equipment, and the method comprises the steps: obtaining target business data from a preset business link, and storing the target business data in a preset database; obtaining total business data from a preset business link, and comparing the total business data with target business data stored in a preset database to obtain a comparison result; if the comparison result indicates that the target business data in the preset database is abnormal, performing data compensation on the target business data in the preset database to enable the target business data in the preset database to be consistent with the total business data; and monitoring the newly added service dependency relationship of the preset service link, and storing the service dependency relationship in a preset database to ensure the integrity of the service link. According to the mode, the problem can be analyzed and the position of the problem can be accurately positioned when the preset service link has the problem through the complete service data and the service dependency relationship stored in the preset database, so that the problem positioning and solving efficiency is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a data processing method, device and electronic device. Background Art

[0002] With the popularization of cloud native and microservice technologies, the dependency links of games and Internet applications are becoming increasingly complex. For example, a game usually relies on peripheral services such as permission verification, billing, and shopping malls, as well as data storage, computing and other resources required for its own deployment. Peripheral services further rely on other services. This complex dependency relationship makes it difficult to quickly locate the source of the problem when a problem occurs. In some companies, traditional CMDB (configuration management database) is used to maintain asset information such as machines and computer rooms. However, when a problem occurs, this method can only manually check the monitoring data of each machine one by one, and then correlate and analyze the root cause of the problem. This method is extremely inefficient and will cause a lot of revenue losses during business peak periods. Summary of the invention

[0003] The purpose of the present disclosure is to provide a data processing method, device and electronic device to efficiently maintain service link data so that when problems occur, they can be discovered and solved in time according to the service link data.

[0004] In a first aspect, the present disclosure provides a data processing method, the method comprising: obtaining target business data from a preset business link, and storing the target business data in a preset database; wherein the preset business link is used to provide external services; the preset database provides data storage and data query services; obtaining full business data from the preset business link, and comparing the full business data with the target business data stored in the preset database to obtain a comparison result; wherein the comparison result is used to indicate whether the target business data in the preset database is missing or abnormal relative to the full business data; if the comparison result indicates that the target business data in the preset database is missing or abnormal, data compensation is performed on the target business data in the preset database; monitoring the newly added business dependencies of the preset business link, and storing the business dependencies in the preset database.

[0005] In a second aspect, the present disclosure provides a data processing device, which includes: a data storage module, which is used to obtain target business data from a preset business link and store the target business data in a preset database; wherein the preset business link is used to provide external services; the preset database provides data storage and data query services; a data comparison module, which is used to obtain full business data from the preset business link, and compare the full business data with the target business data stored in the preset database to obtain a comparison result; wherein the comparison result is used to indicate whether the target business data in the preset database is missing or abnormal relative to the full business data; a data compensation module, which is used to compensate the target business data in the preset database if the comparison result indicates that the target business data in the preset database is missing or abnormal; a relationship update module, which is used to monitor the newly added business dependencies in the preset business link, and store the business dependencies in the preset database.

[0006] In a third aspect, the present disclosure provides an electronic device, which includes a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the above-mentioned data processing method.

[0007] In a fourth aspect, the present disclosure provides a computer-readable storage medium storing computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the above-mentioned data processing method.

[0008] The embodiments of the present disclosure bring the following beneficial effects:

[0009] The present disclosure provides a data processing method, device and electronic device, firstly obtaining target business data from a preset business link, and storing the target business data in a preset database; wherein the preset business link is used to provide external services; and then the preset database provides data storage and data query services, so that link problems can be located by querying the data stored in the preset database; then obtaining full business data from the preset business link, and comparing the full business data with the target business data stored in the preset database to obtain a comparison result; wherein the comparison result is used to indicate whether the target business data in the preset database is missing or abnormal relative to the full business data; if the comparison result indicates that the target business data in the preset database is missing or abnormal, data compensation is performed on the target business data in the preset database to make the target business data in the preset database consistent with the full business data; monitoring the newly added business dependencies of the preset business link, and storing the business dependencies in the preset database to ensure the integrity of the business link. This method can analyze the problem and accurately locate the problem location when a problem occurs in the preset business link through the complete business data and business dependencies stored in the preset database, thereby improving the efficiency of problem location and resolution.

[0010] Other features and advantages of the present disclosure will be set forth in the following description, or some features and advantages may be inferred or unambiguously determined from the description, or may be learned by implementing the above-mentioned technology of the present disclosure.

[0011] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, the following specifically cites preferred implementation modes and describes them in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the specific embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the specific embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0013] Figure 1 A flowchart of a data processing method provided by an embodiment of the present disclosure;

[0014] Figure 2 A schematic diagram of an enterprise application service link topology structure provided in an embodiment of the present disclosure;

[0015] Figure 3 An architecture diagram of a service link maintenance system provided by an embodiment of the present disclosure;

[0016] Figure 4A schematic diagram of the structure of a data processing device provided in an embodiment of the present disclosure;

[0017] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure.

[0018] Reference numerals: S201 - database; S202 - data entry channel; S203 - data source; S204 - data quality monitoring; S205 - data dynamic discovery and compensation. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all the embodiments. The components of the embodiments of the present disclosure described and shown in the drawings here can be arranged and designed in various different configurations.

[0020] Therefore, the following detailed description of the embodiments of the present disclosure provided in the accompanying drawings is not intended to limit the scope of the present disclosure claimed for protection, but merely represents selected embodiments of the present disclosure. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present disclosure.

[0021] In some enterprises, traditional CMDB is used to maintain asset information such as machines and computer rooms. This manual maintenance method can neither guarantee the integrity and accuracy of the data, nor does it lack application information deployed on the machine and dependency information between applications. As a result, when problems arise, the only way is to manually check the monitoring data of each machine one by one, and then analyze the root cause of the problem. This is extremely inefficient and will result in a lot of revenue losses during business peak periods.

[0022] Some companies will introduce technologies such as eBPF (Extended Berkeley Packet Filte, a virtual machine technology running in the Linux kernel) on the basis of traditional CMDB to obtain network call information and supplement the call relationship between machines. However, there is still a problem of missing application information deployed on the machine, and the above problems cannot be solved.

[0023] Based on the above problems, the embodiments of the present disclosure provide a data processing method, device and electronic device. The technology can be applied to data storage and monitoring scenarios of various businesses.

[0024] In order to facilitate understanding of the embodiments of the present disclosure, a data processing method provided by the embodiments of the present disclosure is first described in detail. Figure 1As shown, the method includes the following specific steps:

[0025] Step S102, obtaining target business data from a preset business link, and storing the target business data in a preset database; wherein the preset business link is used to provide external services; and the preset database provides data storage and data query services.

[0026] In specific implementation, the above-mentioned preset business link is mainly used to provide various services to the outside world. The services here can be determined according to R&D needs. For example, the service can include but is not limited to permission verification service, gateway service, billing service and mall service. The above-mentioned target business data can include but is not limited to business models, business fields and model relationships. The specific database of the above-mentioned preset database can be determined according to R&D needs. For example, the preset database can be a non-relational database or a relational database. The preset database has the ability of data storage and data query. The preset database stores the data obtained from the preset business link through the data storage capability. When an abnormality occurs in the preset business link, the data query capability of the preset database is used to automatically query the data from the preset database for accurate problem location, so as to reduce the manual consumption of manual problem location.

[0027] The topology of the above preset service links can be determined according to R&D requirements, such as Figure 1 FIG. 1 is a schematic diagram of a topology structure of an enterprise application service link provided by an embodiment of the present disclosure. Figure 1 The business link topology structure is mainly divided into the application layer, PaaS layer, SaaS layer and IaaS layer. The application layer is the external service of the enterprise, which brings revenue to the enterprise. They are usually independent of each other, but rely on the services of the PaaS layer, SaaS layer and IaaS layer; the PaaS layer is usually a platform service encapsulated within the enterprise, which is relied upon by the application layer, such as unified permission verification service, gateway service, billing service, mall service, etc.; the SaaS layer is usually software services provided by the enterprise, such as databases such as MongoDB and MySQL, and middleware such as Redis, which are relied upon by the application layer or PaaS layer S102; the Iaas layer is usually infrastructure services provided by the enterprise, including servers, storage, networks, and virtualization environments, and the IaSS layer is relied upon by the application layer, PaaS layer and SaaS layer.

[0028] Step S104, obtaining full business data from the preset business link, and comparing the full business data with the target business data stored in the preset database to obtain a comparison result; wherein the comparison result is used to indicate whether the target business data in the preset database is missing or abnormal relative to the full business data.

[0029] In the specific implementation, it is necessary not only to store the target business data obtained by the preset business link in the preset database in real time, but also to obtain the full business data (that is, all business data) from the preset business link in real time or regularly, and compare and check the full business data with all the business data stored in the preset database in real time or regularly to determine whether all the business data in the preset link is stored in the preset database, and whether the stored business data is correct and complete.

[0030] Step S106: If the comparison result indicates that the target service data in the preset database is missing or abnormal, data compensation is performed on the target service data in the preset database.

[0031] The business data stored in the preset database contains missing data or abnormal data relative to the full business data. It is necessary to compensate the business data stored in the preset database based on the full business data to make the target business data in the preset database consistent with the full business data, thereby ensuring that the business data stored in the preset database is correct and complete. When a problem occurs in a certain link in the preset business link, the problem can be traced through the business data stored in the preset database.

[0032] Step S108: monitoring the newly added service dependency of the preset service link, and storing the service dependency in a preset database.

[0033] In specific implementation, the present disclosure can also automatically discover the services involved in the network call in the preset business link, and automatically compensate the business dependencies in the preset database to ensure the integrity and timeliness of the business link, so that when problems occur in the preset business link, the problems can be accurately located and analyzed according to the business dependencies.

[0034] A data processing method, device and electronic device provided by the embodiments of the present disclosure can analyze problems and accurately locate the problem when problems occur in the preset business link through the complete business data and business dependencies stored in the preset database, thereby improving the efficiency of problem location and resolution.

[0035] The following embodiments are used to describe the method of acquiring and storing business data.

[0036] Specifically, the specific process of obtaining the target business data from the preset business link and storing the target business data in the preset database may include: receiving the original business data pushed by the preset business link; extracting the target business data from the original business data, and determining whether the target business data meets the preset requirements; wherein the target business data includes a business model, business fields and model relationships; if the preset requirements are met, the target business data is stored in the preset database.

[0037] In specific implementation, the preset business link is the management department where the application and dependent services are located, including PaaS services, SaaS services, IaaS services, etc., which is responsible for calling the data push interface to push business data in real time, and providing a full data acquisition interface for data quality monitoring. The preset business link can push raw data to the data entry channel in real time, so that the data entry channel can extract target business data from the raw business data, and when the target business data meets the preset requirements, the target business data is stored in the preset database through synchronous or asynchronous entry. Among them, the preset requirements can be determined according to R&D needs. For example, the preset requirements can include at least one of the following: verifying that the business field contains the field defined by the model, and the value of the business field must conform to the type defined by the field, such as integer or string.

[0038] In an optional embodiment, the above-mentioned specific process of extracting target business data from original business data and determining whether the target business data meets preset requirements may include: extracting business models and business fields from original business data; determining whether the business fields meet preset model field requirements, and if so, extracting model relationships from original business data according to preset model relationship definitions; and determining whether the model relationships meet preset model relationship requirements.

[0039] In the specific implementation, after receiving the original business data pushed by the preset business link in the data entry channel, it is necessary to preliminarily verify the legitimacy of the original business data, for example, whether the original business data is empty data or sensitive data. If the original business data is legal, extract the business model and business field from the original business data, and verify whether the business field meets the preset model field requirements. The model field requirements can be determined according to R&D requirements, for example, the model field requirements include data types, required fields, unique identifiers, etc. If the business field meets the preset model field requirements, extract the model relationship in the original business data according to the preset model relationship definition, and then verify whether the model relationship meets the preset model relationship requirements. The preset model relationship requirements can be determined according to R&D requirements, for example, the model relationship requirements can be one-to-one, one-to-many, or many-to-many.

[0040] In actual applications, the preset database includes model data corresponding to multiple business models and model relationships between different business models; among them, the model data includes basic fields and business fields, and each business model is configured with a corresponding database interface, which is used to update the model data of the corresponding business model.

[0041] In the specific implementation, at the model definition level, the preset database uses a collection to store all business model definitions, including basic fields and business fields, and each record in the collection represents a business model; another collection is used to store model relationships, and each record in the collection represents the relationship between two business models. Among them, the types of model relationships include one-to-one, one-to-many, and many-to-many. It should be noted that the above business model definition is used to indicate the above preset model field requirements.

[0042] Specifically, the basic fields of a business model may include model name, model code, department, category, unique identifier, maintainer, creation time, and update time, etc. A business model may include multiple business fields, each of which is customized according to the business represented by the model. For example, a business field may include name, code, data type, field grouping, validation rules, default value, and display type, etc. Model relations must also contain fields, which may include relation name, relation code, source model code, destination model code, and relation type, etc. The relation types include one-to-one, one-to-many, many-to-one, and many-to-many.

[0043] Assume that a business model is used to represent a certain type of service or resource, the model relationship represents the dependency between services, and the model instance represents a specific service or resource. For example, if the model is a machine, a specific machine (such as a machine with an IP of 1.1.1.1) is an instance of the model, and the relationship between instances follows the model relationship definition. At the model instance storage level, for each business model, a collection equivalent to the model code is used to store each instance, and the number and format of fields meet the business field definition requirements of the business model. Each time an instance is created, the database automatically assigns a globally unique id, which can be used as the unique identifier of the instance, or one or more fields can be specified as the unique identifier of the instance when the model is defined. If there is an instance dependency relationship (equivalent to the above model relationship) stored, a field equivalent to the code of the dependent model is added based on the business field defined by the model. The value of the relationship field is an array, and the array element is the id of the dependent instance. This is compatible with one-to-one, one-to-many, and many-to-many scenarios.

[0044] Specifically, each business model in the above preset database is configured with a corresponding database interface, which is used to update the model data of the corresponding business model, that is, there is a one-to-one correspondence between the business model and the database interface. For example, the preset database includes two business models, namely model 1 and model 2. Assuming that the code name of model 1 is service_1 and the code name of model 2 is servic_2, the model instance operation APIs corresponding to these two business models (that is, the above database interface) are / api / v1 / service_1 and / api / v1 / service_2, respectively. The model relationship operation APIs corresponding to these two business models are / api / v1 / service_1 / {service_1_id} / service_2 and / api / v1 / service_2 / {service_2_id} / service_1. These two APIs are equivalent, and users can use any one of them to create or delete relationships. Among them, service_1_id and service_2_id are the ids of a specific data record of the model. The preset business link uses the above API to synchronize the instances and relationships of model 1 and model 2 in real time to ensure data integrity and accuracy.

[0045] Based on the above description, the specific process of storing the target business data in the preset database may include: detecting whether the target database interface corresponding to the business model contained in the target business data exists; if it exists, calling the target database interface; if it does not exist, creating a new target database interface; storing the business fields contained in the target business data into the preset database through the target database interface; updating the model relationship between the business models in the preset database based on the model relationship contained in the target business data.

[0046] In the specific implementation, it is first necessary to check whether there is a target database interface corresponding to the business model contained in the currently obtained target business data. If so, directly call the target database interface to store the business fields contained in the target business data into the preset database to update the data in the preset data path; if not, it is necessary to create a new target database interface to store the business fields contained in the target business data into the preset database through the newly created target database interface. Among them, since the collection in the preset database is used to store business fields, the collection name is equivalent to the business model code. For example, if the business model code is service, the corresponding collection in the preset database is also called service. Finally, it is also necessary to update the collection name and id to the relationship field of the associated business model, that is, to update the model relationship between the business models in the preset database based on the model relationship contained in the target business data, which can also be called updating the association relationship between instances.

[0047] In an optional embodiment, the preset database may be a MongoDB database. As a non-relational database, MongoDB supports flexible data models and can store any type of data, which is necessary for the flexible definition of business models. Moreover, MongoDB provides high-performance data storage and query capabilities, uses B-tree as its index structure, and can quickly search and update data, which is crucial for querying business links.

[0048] Use MongoDB collections as a type of business model, limit the fields and types of each business model, and use MongoDB references as dependencies between models (equivalent to the above model relationships).

[0049] The following embodiments are used to describe the method of ensuring data integrity.

[0050] Specifically, the above-mentioned process of obtaining full business data from a preset business link and comparing the full business data with the target business data stored in a preset database to obtain a comparison result may include: obtaining full business data from the preset business chain, and storing the full business data in a preset storage location; obtaining target business data stored in the preset database, and storing the target business data in a preset storage location; and pulling the full business data and the target business data from the preset storage location for full comparison according to the preset data comparison rules to obtain a comparison result.

[0051] In specific implementation, it is necessary to formulate preset data comparison rules (also called business inspection rules) according to business needs to ensure the integrity, accuracy and consistency of the data stored in the preset database. For example, the preset data comparison rules may include but are not limited to one or more of the following: inspection indicators: such as whether the data field types are consistent, whether the data is redundant or missing, etc.; inspection thresholds: such as data accuracy reaching 100%, data integrity reaching 95%, etc.; inspection object: which specific business model to check, whether it is a machine or a database service, etc.; inspection task scheduling time: set the inspection cycle.

[0052] In the specific implementation, you can call the data source interface to regularly obtain the full amount of business data from the preset business link and store it in the preset storage location (for example, Hive module) to ensure the integrity of the data and facilitate subsequent data processing and analysis. Then, regularly obtain the full amount of business data (the business data includes model instances and model relationship data) from the preset database and also store it in the preset storage location; then set the data proofreading job, regularly pull the full amount of business data of the specified service from the preset storage location and the business data stored in the preset database, and perform a full comparison according to the above-mentioned preset data comparison rules to obtain the comparison result.

[0053] If the comparison result indicates that there are abnormal data such as missing entries or missing fields in the preset database, an alarm notification will be issued to allow the administrator to discover data problems in a timely manner; then the data will be manually checked and data compensation will be performed on the business data in the preset database.

[0054] In an optional embodiment, after the target business data in the preset database is compensated for data, the data storage logic corresponding to the preset database can also be corrected based on the problem indicated by the comparison result. Usually, the comparison result is abnormal because the data storage logic corresponding to the preset database is wrong. In order to ensure that similar problems do not recur, the data storage logic needs to be corrected to reduce the occurrence of data storage problems.

[0055] The following embodiments are used to describe a method for ensuring the integrity of a service link.

[0056] Specifically, a monitoring module is deployed in the server corresponding to the above-mentioned preset business link; the specific process of monitoring the newly added business dependencies of the preset business link and storing the business dependencies in the preset database to ensure the integrity of the business link may include: obtaining network call data and obtaining the target IP address from the network call data through the monitoring module; querying the business model associated with the server corresponding to the target IP address from the preset database, and supplementing the model relationship between the business models in the preset data based on the business model corresponding to the target IP address; wherein the model relationship is a business dependency relationship.

[0057] In specific implementation, the specific module corresponding to the above monitoring module can be determined according to the research and development needs. For example, the monitoring module can be eBPF (Extended Berkeley Packet Filter), and other monitoring systems can also be started. Among them, eBPF is a lightweight virtual machine running in the Linux kernel, which can be used for network filtering, performance analysis, security and other purposes; eBPF programs can be dynamically inserted into the kernel to monitor and analyze the operation of the system in real time; eBPF has been introduced since Linux version 3.15, but many advanced features require a higher version of the kernel. In a specific embodiment, the present disclosure can use a kernel of Linux version 4.8 or above.

[0058] Specifically, use C language to write the eBPF program, and then use the Clang / LLVM compiler to compile it into BPF bytecode; use bpftool or libbpf to load the compiled BPF bytecode into the kernel; attach the eBPF program to a specific hook point, such as a network interface or a system call. In this way, whenever this hook point is triggered, the eBPF program will be executed. In addition, the above eBPF program needs to be deployed on all servers of the preset business link to output the network call information of all servers in real time, including the initiating IP and port of the request, the destination IP and port, etc.

[0059] In actual applications, the network call data is first obtained through the eBPF program, and the target IP address corresponding to each network call is obtained from the network call data; then the business model (or business instance) associated with the server corresponding to each target IP address is queried from the preset database, and then the interface of the preset database is called to supplement the model relationship of the business model, which can also be called the association relationship of the business instance.

[0060] In order to facilitate understanding of the embodiments of the present disclosure, Figure 3 An architecture diagram of a business link maintenance system is provided, which is mainly divided into five modules: a database (S201), a data entry channel (S202), a data source (S203), data quality monitoring (S204), and data dynamic discovery and compensation (S205). The business data of the data source (equivalent to the above-mentioned preset business link) is processed by the data entry channel and stored in the database. The data quality module regularly pulls the full amount of business data from the data source and compares and corrects it with the business data stored in the database to ensure the integrity and accuracy of the data stored in the preset database. The data dynamic discovery and compensation module promptly discovers the newly added link dependency and supplements it to the database.

[0061] Specifically, the present disclosure provides a standardized data entry channel to implement a set of standardized business registration and business link data entry interfaces and processes to ensure the accuracy and consistency of business link data, including mechanisms such as data access, data conversion, and data storage to improve data quality and reliability. In addition, it provides real-time data monitoring capabilities, that is, to design a comprehensive data monitoring module to track and complete business links in real time to ensure the integrity, accuracy, and consistency of business links. At the same time, it also provides link automatic discovery and compensation capabilities. This method is based on eBPF technology to automatically discover the business involved in the network call, and automatically compensate for business dependencies to ensure the integrity and timeliness of the business link. In addition to eBPF, real-time network call information of the request can also be obtained through technologies such as Tracing.

[0062] In the disclosed embodiment, through the collaborative work of several modules, enterprises can quickly build application dependency links while ensuring the integrity, accuracy and timeliness of the links. Based on the business links, enterprises can implement more intelligent operation and maintenance scenarios, such as quickly locating the source of problems, quickly discovering the impact of an abnormal service, and improving business stability. Assisting businesses in capacity management and reducing resource costs, etc.

[0063] Corresponding to the above method embodiment, the present disclosure embodiment also provides a data processing device, such as Figure 4 As shown, the device comprises:

[0064] The data storage module 40 is used to obtain target business data from a preset business link and store the target business data in a preset database; wherein the preset business link is used to provide external services; and the preset database provides data storage and data query services.

[0065] The data comparison module 41 is used to obtain full business data from a preset business link, and compare the full business data with the target business data stored in a preset database to obtain a comparison result; wherein the comparison result is used to indicate whether the target business data in the preset database is missing or abnormal relative to the full business data.

[0066] The data compensation module 42 is used to compensate the target service data in the preset database if the comparison result indicates that the target service data in the preset database is missing or abnormal.

[0067] The relationship updating module 43 is used to monitor the newly added service dependency relationship of the preset service link and store the service dependency relationship in a preset database.

[0068] The above-mentioned data processing device can analyze problems and accurately locate the problem location when problems occur in the preset business link through the complete business data and business dependencies stored in the preset database, thereby improving the efficiency of problem location and resolution.

[0069] Specifically, the above-mentioned data storage module 40 is used to: receive the original business data pushed by the preset business link; extract the target business data from the original business data, and determine whether the target business data meets the preset requirements; wherein the target business data includes the business model, business fields and model relationships; if the preset requirements are met, the target business data is stored in the preset database.

[0070] Furthermore, the above-mentioned data storage module 40 is used to: extract business models and business fields from the original business data; determine whether the business fields meet the preset model field requirements, and if so, extract model relationships from the original business data according to the preset model relationship definition; and determine whether the model relationships meet the preset model relationship requirements.

[0071] In the specific implementation, the above-mentioned preset database includes model data corresponding to multiple business models, as well as model relationships between different business models; wherein the model data includes basic fields and business fields, and each business model is configured with a corresponding database interface, which is used to update the model data of the corresponding business model.

[0072] Furthermore, the above-mentioned data storage module 40 is used to: detect whether a target database interface corresponding to the business model contained in the target business data exists; if it exists, call the target database interface; if it does not exist, create a new target database interface; store the business fields contained in the target business data into a preset database through the target database interface; update the model relationship between the business models in the preset database based on the model relationship contained in the target business data.

[0073] Furthermore, the above-mentioned data comparison module 41 is used to: obtain full business data from the preset business chain, and store the full business data in a preset storage location; obtain target business data stored in the preset database, and store the target business data in a preset storage location; according to the preset data comparison rules, pull the full business data and the target business data from the preset storage location for full comparison to obtain the comparison result.

[0074] Furthermore, the above-mentioned device also includes a logic correction module, which is used to: after data compensation is performed on the target business data in the preset database, based on the problems indicated by the comparison results, correct the data storage logic corresponding to the preset database.

[0075] Furthermore, a monitoring module is deployed in the server corresponding to the above-mentioned preset business link; based on this, the above-mentioned relationship update module 43 is used to: obtain network call data and obtain the target IP address from the network call data through the monitoring module; query the business model associated with the server corresponding to the target IP address from the preset database, and based on the business model corresponding to the target IP address, supplement the model relationship between the business models in the preset data; wherein the model relationship is a business dependency relationship.

[0076] The data processing device provided in the embodiment of the present disclosure has the same implementation principle and technical effects as those of the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding contents in the aforementioned method embodiment.

[0077] The present disclosure also provides an electronic device, such as Figure 5 As shown, the electronic device includes a processor and a memory, the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the above data processing method.

[0078] Specifically, the above-mentioned data processing method includes: obtaining target business data from a preset business link, and storing the target business data in a preset database; wherein the preset business link is used to provide external services; the preset database provides data storage and data query services; obtaining full business data from the preset business link, and comparing the full business data with the target business data stored in the preset database to obtain a comparison result; wherein the comparison result is used to indicate whether the target business data in the preset database is missing or abnormal relative to the full business data; if the comparison result indicates that the target business data in the preset database is missing or abnormal, data compensation is performed on the target business data in the preset database; monitoring the newly added business dependencies of the preset business link, and storing the business dependencies in the preset database.

[0079] The above data processing method can analyze problems and accurately locate the problem location when problems occur in the preset business link through the complete business data and business dependencies stored in the preset database, thereby improving the efficiency of problem location and resolution.

[0080] In an optional embodiment, the above-mentioned steps of obtaining target business data from a preset business link and storing the target business data in a preset database include: receiving original business data pushed by the preset business link; extracting target business data from the original business data, and determining whether the target business data meets preset requirements; wherein the target business data includes a business model, business fields, and model relationships; if the preset requirements are met, storing the target business data in the preset database.

[0081] In an optional embodiment, the above-mentioned steps of extracting target business data from the original business data and determining whether the target business data meets the preset requirements include: extracting business models and business fields from the original business data; determining whether the business fields meet the preset model field requirements, and if so, extracting model relationships from the original business data according to the preset model relationship definition; and determining whether the model relationships meet the preset model relationship requirements.

[0082] In an optional embodiment, the above-mentioned preset database includes model data corresponding to multiple business models, and model relationships between different business models; wherein the model data includes basic fields and business fields, and each business model is configured with a corresponding database interface, which is used to update the model data of the corresponding business model.

[0083] In an optional embodiment, the above-mentioned step of storing the target business data in a preset database includes: detecting whether a target database interface corresponding to the business model contained in the target business data exists; if it exists, calling the target database interface; if it does not exist, creating a new target database interface; storing the business fields contained in the target business data into the preset database through the target database interface; and updating the model relationship between the business models in the preset database based on the model relationship contained in the target business data.

[0084] In an optional embodiment, the above-mentioned step of obtaining full business data from a preset business link, and comparing the full business data with the target business data stored in a preset database to obtain a comparison result includes: obtaining full business data from the preset business chain, and storing the full business data in a preset storage location; obtaining target business data stored in the preset database, and storing the target business data in a preset storage location; according to preset data comparison rules, pulling the full business data and the target business data from the preset storage location for full comparison to obtain a comparison result.

[0085] In an optional embodiment, after the step of performing data compensation on the target business data in the preset database, the method further includes: based on the problem indicated by the comparison result, correcting the data storage logic corresponding to the preset database.

[0086] In an optional embodiment, a monitoring module is deployed in the server corresponding to the above-mentioned preset business link; based on this, the above-mentioned steps of monitoring the newly added business dependencies of the preset business link and storing the business dependencies in the preset database to ensure the integrity of the business link include: obtaining network call data and obtaining the target IP address from the network call data through the monitoring module; querying the business model associated with the server corresponding to the target IP address from the preset database, and supplementing the model relationship between the business models in the preset data based on the business model corresponding to the target IP address; wherein the model relationship is a business dependency relationship.

[0087] Further, Figure 5 The electronic device shown further includes a bus 102 and a communication interface 103 , and the processor 101 , the communication interface 103 and the memory 100 are connected via the bus 102 .

[0088] The memory 100 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 103 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used. The bus 102 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0089] The processor 101 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 101. The above processor 101 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The disclosed methods, steps and logic block diagrams in the embodiments of the present disclosure can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present disclosure can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 100, and the processor 101 reads the information in the memory 100 and completes the steps of the method of the above embodiment in combination with its hardware.

[0090] The embodiments of the present disclosure also provide a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the above-mentioned data processing method. The specific implementation can be found in the method embodiment, which will not be repeated here.

[0091] Specifically, the above-mentioned data processing method includes: obtaining target business data from a preset business link, and storing the target business data in a preset database; wherein the preset business link is used to provide external services; the preset database provides data storage and data query services; obtaining full business data from the preset business link, and comparing the full business data with the target business data stored in the preset database to obtain a comparison result; wherein the comparison result is used to indicate whether the target business data in the preset database is missing or abnormal relative to the full business data; if the comparison result indicates that the target business data in the preset database is missing or abnormal, data compensation is performed on the target business data in the preset database; monitoring the newly added business dependencies of the preset business link, and storing the business dependencies in the preset database.

[0092] The above data processing method can analyze problems and accurately locate the problem location when problems occur in the preset business link through the complete business data and business dependencies stored in the preset database, thereby improving the efficiency of problem location and resolution.

[0093] In an optional embodiment, the above-mentioned steps of obtaining target business data from a preset business link and storing the target business data in a preset database include: receiving original business data pushed by the preset business link; extracting target business data from the original business data, and determining whether the target business data meets preset requirements; wherein the target business data includes a business model, business fields, and model relationships; if the preset requirements are met, storing the target business data in the preset database.

[0094] In an optional embodiment, the above-mentioned steps of extracting target business data from the original business data and determining whether the target business data meets the preset requirements include: extracting business models and business fields from the original business data; determining whether the business fields meet the preset model field requirements, and if so, extracting model relationships from the original business data according to the preset model relationship definition; and determining whether the model relationships meet the preset model relationship requirements.

[0095] In an optional embodiment, the above-mentioned preset database includes model data corresponding to multiple business models, and model relationships between different business models; wherein the model data includes basic fields and business fields, and each business model is configured with a corresponding database interface, which is used to update the model data of the corresponding business model.

[0096] In an optional embodiment, the above-mentioned step of storing the target business data in a preset database includes: detecting whether a target database interface corresponding to the business model contained in the target business data exists; if it exists, calling the target database interface; if it does not exist, creating a new target database interface; storing the business fields contained in the target business data into the preset database through the target database interface; and updating the model relationship between the business models in the preset database based on the model relationship contained in the target business data.

[0097] In an optional embodiment, the above-mentioned step of obtaining full business data from a preset business link, and comparing the full business data with the target business data stored in a preset database to obtain a comparison result includes: obtaining full business data from the preset business chain, and storing the full business data in a preset storage location; obtaining target business data stored in the preset database, and storing the target business data in a preset storage location; according to preset data comparison rules, pulling the full business data and the target business data from the preset storage location for full comparison to obtain a comparison result.

[0098] In an optional embodiment, after the step of performing data compensation on the target business data in the preset database, the method further includes: based on the problem indicated by the comparison result, correcting the data storage logic corresponding to the preset database.

[0099] In an optional embodiment, a monitoring module is deployed in the server corresponding to the above-mentioned preset business link; based on this, the above-mentioned steps of monitoring the newly added business dependencies of the preset business link and storing the business dependencies in the preset database to ensure the integrity of the business link include: obtaining network call data and obtaining the target IP address from the network call data through the monitoring module; querying the business model associated with the server corresponding to the target IP address from the preset database, and supplementing the model relationship between the business models in the preset data based on the business model corresponding to the target IP address; wherein the model relationship is a business dependency relationship.

[0100] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a terminal device, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0101] In the description of the present disclosure, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present disclosure and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation of the present disclosure. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0102] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The protection scope of the present disclosure is not limited thereto. Although the present disclosure is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed in the present disclosure, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A data processing method, characterized in that: The method comprises: Obtain target business data from a preset business link, and store the target business data in a preset database; wherein the preset business link is used to provide external services; the preset database provides data storage and data query services; Acquire full service data from the preset service link, and compare the full service data with the target service data stored in the preset database to obtain a comparison result; wherein the comparison result is used to indicate whether the target service data in the preset database is missing or abnormal relative to the full service data; If the comparison result indicates that the target service data in the preset database is missing or abnormal, data compensation is performed on the target service data in the preset database; The newly added service dependency of the preset service link is monitored, and the service dependency is stored in the preset database.

2. The method according to claim 1, characterized in that: The step of acquiring target service data from a preset service link and storing the target service data in a preset database includes: Receiving the original service data pushed by the preset service link; Extracting target business data from the original business data, and determining whether the target business data meets preset requirements; wherein the target business data includes a business model, business fields, and model relationships; If the preset requirements are met, the target service data is stored in a preset database.

3. The method according to claim 2, characterized in that The step of extracting target business data from the original business data and determining whether the target business data meets preset requirements includes: Extracting a business model and business fields from the original business data; Determine whether the business field meets the preset model field requirements, and if so, extract the model relationship from the original business data according to the preset model relationship definition; Determine whether the model relationship meets the preset model relationship requirements.

4. The method according to claim 2, characterized in that: The preset database includes model data corresponding to multiple business models and model relationships between different business models; wherein the model data includes basic fields and business fields, and each business model is configured with a corresponding database interface, and the database interface is used to update the model data of the corresponding business model.

5. The method according to claim 4, characterized in that The step of storing the target business data in a preset database includes: Detect whether a target database interface corresponding to the business model included in the target business data exists; if so, call the target database interface; if not, create a new target database interface; Storing the business fields included in the target business data into the preset database through the target database interface; The model relationships between the business models in the preset database are updated based on the model relationships included in the target business data.

6. The method according to claim 1, characterized in that The step of acquiring full service data from the preset service link and comparing the full service data with target service data stored in the preset database to obtain a comparison result includes: Acquire full service data from the preset service chain, and store the full service data in a preset storage location; Acquire the target service data stored in the preset database, and store the target service data in the preset storage location; According to the preset data comparison rule, the full amount of business data and the target business data are pulled from the preset storage location for full comparison to obtain a comparison result.

7. The method according to claim 1, characterized in that After the step of performing data compensation on the target service data in the preset database, the method further includes: Based on the problem indicated by the comparison result, the data storage logic corresponding to the preset database is corrected.

8. The method according to claim 1, characterized in that A monitoring module is deployed in the server corresponding to the preset service link; The step of monitoring the newly added service dependency of the preset service link and storing the service dependency in the preset database to ensure the integrity of the service link includes: Obtaining network call data and obtaining a target IP address from the network call data through the monitoring module; The business model associated with the server corresponding to the target IP address is queried from the preset database, and based on the business model corresponding to the target IP address, the model relationship between the business models in the preset data is supplemented; wherein the model relationship is the business dependency relationship.

9. A data processing device, characterized in that: The device comprises: A data storage module, used to obtain target business data from a preset business link and store the target business data in a preset database; wherein the preset business link is used to provide external services; the preset database provides data storage and data query services; A data comparison module, used to obtain full service data from the preset service link, and compare the full service data with the target service data stored in the preset database to obtain a comparison result; wherein the comparison result is used to indicate whether the target service data in the preset database is missing or abnormal relative to the full service data; A data compensation module, configured to compensate the target service data in the preset database if the comparison result indicates that the target service data in the preset database is missing or abnormal; The relationship updating module is used to monitor the newly added service dependency relationship of the preset service link and store the service dependency relationship in the preset database.

10. An electronic device, characterized in that: The electronic device includes a processor and a memory, the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the data processing method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the data processing method according to any one of claims 1 to 8.

Images