Data processing methods and related products

Abstract

The embodiment of the present application provides a data processing method and related products, wherein the method includes: in response to a data processing request from an external system, synchronizing a data source that meets a preset specification from an internal system through an adaptation center, the data source is used by the adaptation center to determine the business scenario corresponding to the data processing request; cleaning the data source according to the business scenario to obtain a target data source, and sending the target data source to the external system. The use of the embodiment of the present application is conducive to improving the accuracy of data acquisition.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method and related products. Background Art

[0002] At present, when many enterprises develop to a certain stage, internal departments often introduce multiple sets of software from different service providers to facilitate them according to business needs. The data between departments are relatively independent, but their businesses are interconnected and collaborative. At this time, in order to make the data of each system flow, it becomes a difficult problem to connect the data. Generally, it is necessary to clarify the business scenario with external manufacturers first, obtain external data, investigate internal business scenarios, and carry out targeted customized development. This method has low compatibility of solutions, and it takes a lot of time and manpower to develop each time, which has high development costs. Summary of the invention

[0003] The embodiments of the present application provide a data processing method and related products, which do not require customized development, are conducive to saving development costs, and are conducive to improving data processing efficiency.

[0004] A first aspect of an embodiment of the present application provides a data processing method, which is applied to a data processing system, wherein the data processing system includes an external system, an adaptation center, and an internal system, including:

[0005] In response to a data processing request from an external system, a data source that meets a preset specification is synchronized from the internal system through the adaptation center, and the data source is used by the adaptation center to determine the business scenario corresponding to the data processing request;

[0006] The data source is cleaned according to the business scenario to obtain a target data source, and the target data source is sent to the external system.

[0007] A second aspect of the present application provides a data processing device, which is applied to a data processing system, wherein the data processing system includes an external system, an adaptation center, and an internal system. The device includes: a synchronization unit and a data cleaning unit, wherein:

[0008] The synchronization unit is used to respond to a data processing request from an external system and synchronize a data source that meets a preset specification from the internal system through the adaptation center, wherein the data source is used by the adaptation center to determine a business scenario corresponding to the data processing request;

[0009] The data cleaning unit is used to clean the data source according to the business scenario to obtain a target data source, and send the target data source to the external system.

[0010] A third aspect of the present application provides an electronic device, comprising: a processor and a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the processor, and the programs include instructions for some or all of the steps described in the first aspect.

[0011] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program, wherein the computer program enables a computer to execute instructions of some or all of the steps described in the first aspect of the embodiment of the present application.

[0012] A fifth aspect of the embodiments of the present application provides a computer program product, wherein the computer program product includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute some or all of the steps described in the first aspect of the embodiments of the present application. The computer program product may be a software installation package.

[0013] Implementing the embodiments of the present application has the following beneficial effects:

[0014] It can be seen that the data processing method and related products described in the embodiments of the present application can respond to data processing requests from external systems, and synchronize data sources that meet preset specifications from the internal system through the adaptation center, and the data source is used by the adaptation center to determine the business scenario corresponding to the data processing request; the data source is cleaned according to the business scenario to obtain the target data source, and the target data source is sent to the external system. In this way, when it is necessary to connect with an external system, there is no need to care about and be familiar with the business scenarios of this system, and customized development is not required, which is conducive to saving development costs; the data source can be directly obtained from the cloud system through the adaptation center according to the business scenario to reduce customized development of data or business scenarios. Finally, data cleaning is performed on the data source, which is conducive to improving the accuracy of data acquisition. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0016] Figure 1A It is a schematic diagram of the system architecture of a data processing method provided in an embodiment of the present application;

[0017] Figure 1BIt is a schematic diagram of the system architecture of a data processing method provided in an embodiment of the present application;

[0018] Figure 1C It is a schematic diagram of an embodiment flow of a data processing method provided in an embodiment of the present application;

[0019] Figure 2 It is a schematic diagram of an embodiment flow of a data processing method provided in an embodiment of the present application;

[0020] Figure 3A is a schematic diagram of the structure of an embodiment of a data processing device provided in an embodiment of the present application;

[0021] Figure 3B is a schematic diagram of the structure of an embodiment of a data processing device provided in an embodiment of the present application;

[0022] Figure 4 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0024] The terms "first", "second", "third" and "fourth" etc. in the specification and claims of the present application and the drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0025] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The presentation of the phrase at various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0026] The electronic devices involved in the embodiments of the present application may include various handheld devices with wireless communication functions, electronic devices, wearable devices, computing devices or other processing devices connected to a wireless modem, as well as various forms of user equipment (UE), mobile stations (MS), terminal devices, etc. For the convenience of description, the above-mentioned devices are collectively referred to as electronic devices. In the present application, the electronic device may also be a server.

[0027] The embodiments of the present application are described in detail below.

[0028] See also Figure 1A , Figure 1A Schematic diagram of the system architecture of a data processing method provided in an embodiment of the present application. Figure 1A As shown, the system architecture includes: an electronic device 110 and a cloud server (or cloud or cloud system, etc.) 120.

[0029] In order to ensure the information security of user information (for example, customer personal information, customer ordering information, housing information, order payment information, consultant information, etc., not limited here), or to save storage space of electronic devices, data of different business scenarios can be stored in the cloud server 120, and can be obtained from the above-mentioned cloud server when needed.

[0030] The business scenarios may include at least one of the following: customer data binding, customer data synchronization, transaction data integration, customer data update, house source information, subscription or transaction information, consultant information query, order information query / update, etc., without limitation. The business scenarios may include different businesses such as processing customer information, processing order information, binding customers and houses, updating transaction data, binding consultants and customers in the real estate system, and each business is a corresponding business process including complex processing logic.

[0031] For example, if the above business scenario includes the binding business of customer information, it generally includes the following processes: add a new subscription / contract document--determine whether there is a house in the cloud system--if so, determine whether there is a current customer in the cloud system--if so, update the customer information--determine whether there is a current order in the cloud system--if so, determine whether the order is a subscription / contract type--determine the subscription time and contract time, and add a new order; among which, when it is determined that there is no current order in the above cloud system, it can be judged whether the document operation time is greater than the order time, and when it is determined that it is greater than the order time, determine whether the order is a contract or subscription situation. There will be different processing mechanisms for different situations, which will not be repeated here. As can be seen from the above process, in this business scenario, there are multiple logical judgments, and the data processing and acquisition operations are complex. The binding business of customer information alone involves the linkage of multiple business scenarios such as order information query and customer information update, and involves multiple acquisitions of data for different business scenarios, which is not conducive to the rapid binding of customer information. In addition, for different business scenarios, new manufacturers may be involved. If a separate program must be developed for each access process, the manpower and time invested will increase, thereby increasing the difficulty of development.

[0032] like Figure 1B As shown, it is a schematic diagram of a system architecture of a data processing method, which may include: a data processing system and a cloud system (cloud server), wherein the data processing system includes: an internal system, an external system and an adaptation center. For the external system, different business scenarios may be encapsulated into a series of adaptation interfaces, and the adaptation interfaces may be provided to the external system.

[0033] Among them, the above-mentioned adaptation center can be used to perform data standardization processing on various types of original data corresponding to different business scenarios according to preset specifications, and write them into the cloud system through the adaptation center.

[0034] Among them, the above-mentioned internal system may refer to the system corresponding to the local database corresponding to the external system, which can be used to store various business logics and underlying data constructed or set by this system; of course, a large amount of user-level data can be stored in the cloud system.

[0035] Among them, the above-mentioned cloud system can be used to store various original data corresponding to various business scenarios.

[0036] Therefore, by Figure 1BThe system can respond to data processing requests from external systems, and synchronize data sources that meet preset specifications from the internal system through the adaptation center. The data source is used by the adaptation center to determine the business scenario corresponding to the data processing request; the data source is cleaned according to the business scenario to obtain the target data source, and the target data source is sent to the external system. In this way, each system can be decoupled from each other, and customized development is not required. By encapsulating business interfaces and separating data mapping, the efficiency of data connection is improved, which is conducive to saving development costs. And when it is necessary to connect with an external system, there is no need to care about and be familiar with the business scenarios of this system. The data source can be directly obtained from the cloud system through the adaptation center according to the business scenario to reduce customized development of data or business scenarios. Finally, data cleaning is performed on the data source, which is conducive to improving the accuracy of data acquisition.

[0037] Consistent with the above, see Figure 1C , is a flow chart of an embodiment of a data processing method provided in an embodiment of the present application. Figure 1A The electronic device shown in the figure, the data processing method described in this embodiment includes the following steps:

[0038] 101. In response to a data processing request from an external system, a data source that meets preset specifications is synchronized from the internal system through an adaptation center, and the data source is used by the adaptation center to determine a business scenario corresponding to the data processing request.

[0039] The above preset specifications can be set by the user or by the system default, and are not limited here. The preset specifications can be set according to different business scenarios. For example, for different business scenarios, the document format of the data to be obtained is different, and the data source that meets the preset specifications can refer to the data source composed of all data in the document format that meets the business scenario.

[0040] The above data source may include at least one of the following: image data, text data, video data, chart data, etc., which are not limited here.

[0041] Among them, the above data processing request is initiated by the corresponding personnel of the external system and can be used to access various data in the internal system, which is helpful to meet the needs of various business scenarios.

[0042] In a possible example, before synchronizing the data source that meets the preset specifications from the internal system through the adaptation center, the method also includes: determining the data interface corresponding to the data processing request; determining the target interface type of the data interface; determining the target data synchronization method corresponding to the target interface type based on the mapping relationship between the preset interface type and the data synchronization method; and executing the step of synchronizing the data source that meets the preset specifications from the internal system through the adaptation center based on the target data synchronization method.

[0043] The above preset interface type can be set by the user or by system default, and is not limited here. The interface type can be determined according to different business scenarios, and the interface type can be preset for different data synchronization methods in advance. The data synchronization method can include at least one of the following: interface synchronization, scheduled task synchronization, or message queue synchronization.

[0044] It can be seen that in this application, the data synchronization method is not limited. No matter which data synchronization method the external system supports, this system can provide support to achieve the acquisition of the data source, and the external system does not need to care about its corresponding data interface to obtain the data source.

[0045] In a possible example, synchronizing the data source that meets the preset specifications from the cloud system through the adaptation center includes: determining the data field carried in the data processing request of the external system; determining the data format corresponding to the data field; obtaining preset data content corresponding to the data format from the cloud system through the adaptation center; constructing a first dictionary space according to the preset data content; processing the first dictionary space to obtain multiple sub-first dictionary spaces, and the data sources corresponding to the multiple sub-first dictionary spaces are the data sources that meet the preset specifications.

[0046] Among them, dictionary mapping relationships can be set in advance for data fields corresponding to different business scenarios.

[0047] Among them, the cloud system can store a variety of raw data, and the raw data can be classified according to the data format in advance. Different business scenarios can correspond to different data formats. Then, according to the data format corresponding to the data field, the preset data content corresponding to the data format, that is, the pre-stored raw data, can be obtained from the cloud system.

[0048] Furthermore, the preset data content can be constructed into a first dictionary space, which is a spatial structure. The first dictionary space can be used to map to specific preset data content. The first dictionary space occupies a smaller space than all the preset data content. Constructing the first dictionary space is conducive to quickly locating the data desired by the external system.

[0049] Furthermore, since the first dictionary space is constructed by all the preset data contents, when querying data specifically, the space consumption is very large, which will waste resources. Therefore, the first dictionary space can be processed by dimension reduction (for example, principal component analysis algorithm (PCA) etc.) to obtain the first dictionary space after dimension reduction, and the first dictionary space after dimension reduction can retain more characteristics of the preset data contents. Furthermore, the first dictionary space can be spatially split to obtain multiple sub-first dictionary spaces of the same dimension; further, the data sources corresponding to the multiple sub-first dictionary spaces can be determined as data sources that meet the preset specifications.

[0050] It can be seen that in this example, the data format constraints can be used to meet the standardized data source to achieve the unification of the same business scenario data, which is conducive to protecting the data security of other business scenario data. In addition, the dictionary space can be constructed for the obtained preset data content, and dimensionality reduction processing can be performed, which is conducive to improving the data matching efficiency in the subsequent data processing process.

[0051] 102. Perform data cleansing on the data source according to the business scenario to obtain a target data source, and send the target data source to the external system.

[0052] Among them, the above-mentioned data cleaning method can be set by the user or by system default, and is not limited here; different data cleaning methods can be set according to business scenarios to adapt to the data types of different business scenarios, which is conducive to improving data cleaning efficiency.

[0053] Among them, the data cleaning method may include at least one of the following: the data processing method for data cleaning may include at least one of the following processing operations: data type screening, similarity threshold screening, feature spacing screening, etc.; for example, in a specific implementation, for the image data source, two processing methods, data type screening and similarity threshold screening, can be adopted, which may specifically include the following operation steps: data type screening is performed on the image data set to obtain the first source data that meets the desired data type, and then, the data set is processed with a similarity threshold, that is, image similarity matching is performed, and images corresponding to similarities less than a preset threshold (which can be set by the user or by the system default, and is not limited here) are selected to obtain the final desired target source data, thereby removing redundant data in the image data set.

[0054] Optionally, when performing data cleaning on the source data, different data cleaning methods can be selected, and there is a correlation between the data cleaning methods. The correlation of the data cleaning methods can be determined according to the data type of the source data, thereby making a sequence processing relationship exist between different data cleaning methods. This is conducive to obtaining data that meets the standards and improving the accuracy of data cleaning to remove unwanted redundant data.

[0055] In a possible example, data cleaning is performed on the data source according to the business scenario to obtain a target data source, including: constructing a second dictionary space according to the data field; calculating the distance between the second dictionary space and each of the multiple sub-first dictionary spaces to obtain multiple space distances; selecting multiple target space distances greater than or equal to a preset distance from the multiple space distances; determining the target sub-first dictionary space corresponding to each of the target space distances to obtain multiple target sub-first dictionary spaces; and determining the target data source according to the business scenario and the multiple target sub-first dictionary spaces.

[0056] Among them, the above preset distance can be set by the user or by the system default, which is not limited here. The smaller the spatial distance, the higher the data similarity between the two. Conversely, the larger the spatial distance, the lower the data similarity between the two. The preset distance can be set according to the business needs of the business scenario; specifically, three ranges can be set for the preset distance, namely the first range [a1, b1), the second range [b1, b2) and the third range [b2, a2], b1> a1, b2> b1, a2> b2; if the business demand is to obtain more and more data sources, a value can be randomly selected in the first range as the preset distance to obtain more data source cardinality. If the business demand is to obtain a data source that is neither too much nor too little, a value can be randomly selected in the second range as the preset distance. Compared with the other two cases, the preset distance in this case is the median. If the business demand is to obtain a small data source with high precision, a value can be randomly selected in the third range as the preset distance.

[0057] It should be noted that, in this example, the data sources corresponding to the multiple target sub-first dictionary spaces can be directly used as the target data source.

[0058] It can be seen that in this example, for example, when the business scenario corresponding to the data processing request corresponding to the external system is a high-precision data screening requirement, the similarity between the two dictionary spaces can be determined by the spatial distance and the preset distance, thereby screening out multiple target sub-first dictionaries with high similarity to the data fields in the data processing request corresponding to the external system, which is conducive to the subsequent acquisition of data sources that meet the preset specifications. On the other hand, when it is necessary to connect with an external system, there is no need to care about and be familiar with the business scenarios of this system. It is only necessary to establish a dictionary mapping relationship with the data fields of the external system through the interface provided by this example, which is conducive to improving the efficiency of data connection and data acquisition.

[0059] In a possible example, determining the target data source according to the business scenario and the multiple target sub-second dictionary spaces includes: determining a first privacy coefficient corresponding to the business scenario; determining a data source to be cleaned corresponding to the multiple target sub-first dictionary spaces; determining a second privacy coefficient corresponding to the external system according to a preset mapping relationship between the external system and the privacy coefficient; optimizing the first privacy coefficient by the second privacy coefficient to obtain a target privacy coefficient; determining multiple groups of data related to the business scenario from the data source to be cleaned; generating a data matrix corresponding to each group of data according to the multiple groups of data; generating a privacy matrix according to the target privacy coefficient; performing data cleaning on each data matrix according to the privacy matrix to obtain target data corresponding to each group of data, and the multiple target data constitute the target data source.

[0060] In this example, since the external system may not have sufficient permissions when accessing the internal system, different business scenarios require different levels of data protection. Therefore, the data source actually returned to the external system needs to be cleansed again to prevent the leakage of private data.

[0061] In a specific implementation, the first privacy coefficient corresponding to the business scenario can be determined, and the first privacy coefficient is used to characterize the privacy protection strength of the business scenario. The more privacy protection is needed for the business scenario, the higher the corresponding first privacy coefficient is. For example, compared with the scenario of obtaining customer information and the scenario of obtaining consultant information, the privacy of the scenario of obtaining customer information is obviously higher than that of obtaining consultant information, so the corresponding first privacy coefficient is larger.

[0062] Since the data type or business type of the data obtained in the same business scenario is more than one, the above-mentioned first privacy coefficient is in principle multiple, and each first privacy coefficient corresponds to a business type or data type. If the business scenario is to obtain customer information, the data type or business type may include at least one of the following: voiceprint information, age information, face information, ID card information, occupation information, etc., which are not limited here.

[0063] For another example, if the business scenario is to obtain customer information, different external systems have different permissions to obtain the types of customer information (including: voiceprint information, age information, facial information, ID card information, occupational information, etc., not limited here). Therefore, a mapping relationship between the external system and the privacy coefficient can be preset, and then the higher the access permission corresponding to the external system, the smaller the privacy coefficient.

[0064] Furthermore, the first privacy coefficient corresponding to the business scenario can be adjusted according to the second privacy coefficient corresponding to the external system. In general, the specific adjustment method is: target privacy coefficient = i*first privacy coefficient + j*second privacy coefficient; i+j=1. For the privacy protection of internal system data, i can be set to <j。

[0065] Furthermore, the data corresponding to the multiple target sub-first dictionary spaces obtained after the above-mentioned first data cleaning can be used as the data source to be cleaned, and the data source to be cleaned can be split according to the business scenario. Since the relevance of each group of data to the business scenario is different, completely irrelevant data can be removed to obtain multiple groups of data.

[0066] In this example, the business scenario may also include multiple (two or more), but its specific processing method is consistent with this application and will not be repeated here.

[0067] Finally, a data matrix can be generated based on multiple groups of data, each group of data corresponds to a data matrix; and the above target privacy coefficient is used to generate a privacy matrix. Furthermore, each group of data matrices can be perturbed according to the privacy matrix to obtain the target data corresponding to each group of data, and multiple target data can be combined to obtain the target data source. Among them, the specific perturbation processing method is not limited here. The purpose of perturbation processing is to hide part of the data in multiple groups of data. The larger the target privacy coefficient corresponding to the data, the greater the intensity of the perturbation processing, that is, the greater the noise corresponding to the data.

[0068] It can be seen that in this example, some business scenarios require further data cleansing of the acquired data source so that the data obtained by the external system complies with its corresponding privacy permissions, thereby achieving privacy protection of the data in the internal system, which is conducive to improving data security.

[0069] It can be seen that the data processing method provided by the embodiment of the present application can respond to data processing requests from external systems, and synchronize data sources that meet preset specifications from the internal system through the adaptation center, and the data source is used by the adaptation center to determine the business scenario corresponding to the data processing request; the data source is cleaned according to the business scenario to obtain the target data source, and the target data source is sent to the external system. In this way, when it is necessary to connect with an external system, there is no need to care about and be familiar with the business scenarios of this system, and customized development is not required, which is conducive to saving development costs; the data source can be directly obtained from the cloud system through the adaptation center according to the business scenario to reduce customized development of data or business scenarios. Finally, data cleaning is performed on the data source, which is conducive to improving the accuracy of data acquisition.

[0070] Consistent with the above, see Figure 2 , is a flow chart of an embodiment of a data processing method provided in an embodiment of the present application. The data processing method described in this embodiment is applied to an electronic device and includes the following steps:

[0071] 201. In response to a data processing request from an external system, determine a data field carried in the data processing request from the external system.

[0072] 202. Determine a data format corresponding to the data field.

[0073] 203. Acquire preset data content corresponding to the data format from the cloud system through the adaptation center.

[0074] 204. Construct a first dictionary space according to the preset data content.

[0075] 205 . Process the first dictionary space to obtain a plurality of sub-first dictionary spaces, where data sources corresponding to the plurality of sub-first dictionary spaces are the data sources that meet a preset specification.

[0076] 206. Construct a second dictionary space according to the data field.

[0077] 207 . Calculate the distance between the second dictionary space and each of the plurality of first sub-dictionary spaces to obtain a plurality of space distances.

[0078] 208. Select a plurality of target spatial distances that are greater than or equal to a preset distance from the plurality of spatial distances.

[0079] 209. Determine a target sub-first dictionary space corresponding to each target space distance to obtain a plurality of target sub-first dictionary spaces.

[0080] 210. Determine the target data source according to the business scenario and the multiple target sub-first dictionary spaces, and send the target data source to the external system.

[0081] Optionally, the detailed description of the above steps 201 to 210 can be referred to Figure 1C The corresponding steps of step 101 to step 102 of the described data processing method will not be repeated here.

[0082] It can be seen that the data processing method provided by the embodiment of the present application is applied to an electronic device, responding to a data processing request from an external system, determining the data field carried in the data processing request of the external system; determining the data format corresponding to the data field; obtaining the preset data content corresponding to the data format from the cloud system through the adaptation center; constructing a first dictionary space according to the preset data content; processing the first dictionary space to obtain multiple sub-first dictionary spaces, and the data sources corresponding to the multiple sub-first dictionary spaces are the data sources that meet the preset specifications; constructing a second dictionary space according to the data field; calculating the distance between the second dictionary space and each sub-first dictionary space in the multiple sub-first dictionary spaces to obtain multiple space distances; selecting multiple target space distances greater than or equal to the preset distance from the multiple space distances; determining the target sub-first dictionary space corresponding to each target space distance to obtain multiple target sub-first dictionary spaces; determining the target data source according to the business scenario and the multiple target sub-first dictionary spaces, and sending the target data source to the external system. Thus, multiple target sub-first dictionaries with high similarity of data fields in the data processing request corresponding to the external system can be screened out, thereby facilitating the subsequent acquisition of a data source that meets the preset specifications. On the other hand, when it is necessary to connect with an external system, you do not need to care about or be familiar with the business scenarios of this system. You only need to use the interface provided by this example to establish a dictionary mapping relationship with the data fields of the external system, which is conducive to improving the efficiency of data connection and data acquisition.

[0083] Consistent with the above, the following is a device for implementing the above data processing method, specifically as follows:

[0084] See also Figure 3A , is a schematic diagram of the structure of an embodiment of a data processing device provided in an embodiment of the present application. The data processing device described in this embodiment is applied to a data processing system, wherein the data processing system includes an external system, an adaptation center, and an internal system. The data processing device includes: a synchronization unit 301 and a data cleaning unit 302, wherein:

[0085] The synchronization unit 301 is used to respond to a data processing request from an external system and synchronize a data source that meets a preset specification from the internal system through the adaptation center, wherein the data source is used by the adaptation center to determine a business scenario corresponding to the data processing request;

[0086] The data cleaning unit 302 is used to clean the data source according to the business scenario to obtain a target data source, and send the target data source to the external system.

[0087] The synchronization unit 301 may be used to implement the method described in step 101 , the data cleaning unit 302 may be used to implement the method described in step 102 , and so on.

[0088] It can be seen that the data processing device described in the embodiment of the present application can respond to data processing requests from external systems, and synchronize data sources that meet preset specifications from the internal system through the adaptation center, and the data source is used by the adaptation center to determine the business scenario corresponding to the data processing request; the data source is cleaned according to the business scenario to obtain the target data source, and the target data source is sent to the external system. In this way, when it is necessary to connect with an external system, there is no need to care about and be familiar with the business scenarios of this system. The data source can be directly obtained from the cloud system through the adaptation center according to the business scenario to reduce customized development of data or business scenarios. Finally, data cleaning of the data source is beneficial to improving the accuracy of data acquisition.

[0089] In a possible example, before synchronizing the data source that meets the preset specification from the internal system through the adaptation center, such as Figure 3B As shown, the data processing device further includes: a determining unit 303, configured to:

[0090] Determining a data interface corresponding to the data processing request;

[0091] Determining a target interface type of the data interface;

[0092] Determine the target data synchronization mode corresponding to the target interface type according to the mapping relationship between the preset interface type and the data synchronization mode;

[0093] According to the target data synchronization mode, the step of synchronizing the data source that meets the preset specification from the internal system through the adaptation center is performed.

[0094] In a possible example, in terms of synchronizing the data source that meets the preset specification from the cloud system through the adaptation center, the synchronization unit 301 is specifically used to:

[0095] Determining a data field carried in the data processing request of the external system;

[0096] Determine the data format corresponding to the data field;

[0097] Acquiring preset data content corresponding to the data format from the cloud system through the adaptation center;

[0098] Constructing a first dictionary space according to the preset data content;

[0099] The first dictionary space is processed to obtain a plurality of sub-first dictionary spaces, and the data sources corresponding to the plurality of sub-first dictionary spaces are the data sources that meet the preset specification.

[0100] In a possible example, in terms of performing data cleaning on the data source according to the business scenario to obtain the target data source, the data cleaning unit 302 is specifically used to:

[0101] Constructing a second dictionary space according to the data field;

[0102] Calculating the distance between the second dictionary space and each of the plurality of first sub-dictionary spaces to obtain a plurality of space distances;

[0103] Selecting a plurality of target spatial distances greater than or equal to a preset distance from the plurality of spatial distances;

[0104] Determine the target sub-first dictionary space corresponding to each target space distance to obtain a plurality of target sub-first dictionary spaces;

[0105] The target data source is determined according to the business scenario and the multiple target sub-first dictionary spaces.

[0106] In a possible example, in determining the target data source according to the business scenario and the multiple target sub-first dictionary spaces, the data cleaning unit 302 is specifically configured to:

[0107] Determining a first privacy coefficient corresponding to the business scenario;

[0108] Determine the to-be-cleaned data sources corresponding to the plurality of target sub-first dictionary spaces;

[0109] Determining a second privacy coefficient corresponding to the external system according to a preset mapping relationship between the external system and the privacy coefficient;

[0110] Optimizing the first privacy coefficient by using the second privacy coefficient to obtain a target privacy coefficient;

[0111] Determining multiple groups of data related to the business scenario from the data source to be cleaned;

[0112] According to the multiple groups of data, a data matrix corresponding to each group of data is generated;

[0113] generating a privacy matrix according to the target privacy coefficient;

[0114] Data cleaning is performed on each data matrix according to the privacy matrix to obtain target data corresponding to each group of data, and the multiple target data constitute the target data source.

[0115] It can be understood that the functions of each program module of the data processing device of this embodiment can be specifically implemented according to the method in the above method embodiment, and its specific implementation process can refer to the relevant description of the above method embodiment, which will not be repeated here.

[0116] Consistent with the above, see Figure 4 , is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device described in this embodiment includes a data processing system, and the data processing system includes an external system, an adaptation center, and an internal system. As shown in the figure, the electronic device includes a processor, a memory, a communication interface, and one or more programs. The one or more programs are stored in the memory and are configured to be executed by the processor. In the embodiment of the present application, the program includes instructions for executing the following steps:

[0117] In response to a data processing request from an external system, a data source that meets a preset specification is synchronized from the internal system through the adaptation center, and the data source is used by the adaptation center to determine the business scenario corresponding to the data processing request;

[0118] The data source is cleaned according to the business scenario to obtain a target data source, and the target data source is sent to the external system.

[0119] It can be seen that the electronic device described in the embodiment of the present application can respond to a data processing request from an external system, and synchronize a data source that meets preset specifications from the internal system through the adaptation center, and the data source is used by the adaptation center to determine the business scenario corresponding to the data processing request; the data source is cleaned according to the business scenario to obtain the target data source, and the target data source is sent to the external system. In this way, when it is necessary to connect with an external system, there is no need to care about and be familiar with the business scenario of this system. The data source can be directly obtained from the cloud system through the adaptation center according to the business scenario to reduce customized development of data or business scenarios. Finally, data cleaning of the data source is beneficial to improving the accuracy of data acquisition.

[0120] In a possible example, before synchronizing the data source that meets the preset specification from the internal system through the adaptation center, the program further includes instructions for executing the following steps:

[0121] Determining a data interface corresponding to the data processing request;

[0122] Determining a target interface type of the data interface;

[0123] Determine the target data synchronization mode corresponding to the target interface type according to the mapping relationship between the preset interface type and the data synchronization mode;

[0124] According to the target data synchronization mode, the step of synchronizing the data source that meets the preset specification from the internal system through the adaptation center is performed.

[0125] In a possible example, in terms of synchronizing the data source that meets the preset specification from the cloud system through the adaptation center, the program includes instructions for performing the following steps:

[0126] Determining a data field carried in the data processing request of the external system;

[0127] Determine the data format corresponding to the data field;

[0128] Acquiring preset data content corresponding to the data format from the cloud system through the adaptation center;

[0129] Constructing a first dictionary space according to the preset data content;

[0130] The first dictionary space is processed to obtain a plurality of sub-first dictionary spaces, and the data sources corresponding to the plurality of sub-first dictionary spaces are the data sources that meet the preset specification.

[0131] In a possible example, in terms of performing data cleansing on the data source according to the business scenario to obtain a target data source, the program includes instructions for executing the following steps:

[0132] Constructing a second dictionary space according to the data field;

[0133] Calculating the distance between the second dictionary space and each of the plurality of first sub-dictionary spaces to obtain a plurality of space distances;

[0134] Selecting a plurality of target spatial distances greater than or equal to a preset distance from the plurality of spatial distances;

[0135] Determine the target sub-first dictionary space corresponding to each target space distance to obtain a plurality of target sub-first dictionary spaces;

[0136] The target data source is determined according to the business scenario and the multiple target sub-first dictionary spaces.

[0137] In a possible example, in determining the target data source according to the business scenario and the multiple target sub-first dictionary spaces, the program includes instructions for performing the following steps:

[0138] Determining a first privacy coefficient corresponding to the business scenario;

[0139] Determine the to-be-cleaned data sources corresponding to the plurality of target sub-first dictionary spaces;

[0140] Determining a second privacy coefficient corresponding to the external system according to a preset mapping relationship between the external system and the privacy coefficient;

[0141] Optimizing the first privacy coefficient by using the second privacy coefficient to obtain a target privacy coefficient;

[0142] Determining multiple groups of data related to the business scenario from the data source to be cleaned;

[0143] According to the multiple groups of data, a data matrix corresponding to each group of data is generated;

[0144] generating a privacy matrix according to the target privacy coefficient;

[0145] Data cleaning is performed on each data matrix according to the privacy matrix to obtain target data corresponding to each group of data, and the multiple target data constitute the target data source.

[0146] An embodiment of the present application also provides a computer storage medium, wherein the computer storage medium may store a program, and when the program is executed, the program includes part or all of the steps of any one of the data processing methods recorded in the above method embodiments.

[0147] Although the present application is described herein in conjunction with various embodiments, in the process of implementing the claimed application, those skilled in the art may understand and implement other variations of the disclosed embodiments by viewing the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other components or steps, and "one" or "an" does not exclude multiple situations. A single processor or other unit may implement several functions listed in a claim. Certain measures are recorded in different dependent claims, but this does not mean that these measures cannot be combined to produce good results.

[0148] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, devices (equipment), or computer program products. Therefore, the present application may adopt the form of complete hardware embodiments, complete software embodiments, or embodiments in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code. The computer program is stored / distributed in a suitable medium, provided together with other hardware or as a part of hardware, and other distribution forms may also be adopted, such as by Internet or other wired or wireless telecommunication systems.

[0149] The present application is described with reference to the flowchart and / or block diagram of the method, device (equipment) and computer program product of the embodiment of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, and the combination of the process and / or box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the function specified in one process or multiple processes in the flowchart and / or one box or multiple boxes in the block diagram.

[0150] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0151] These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0152] Although the present application has been described in conjunction with specific features and embodiments thereof, it is obvious that various modifications and combinations may be made thereto without departing from the spirit and scope of the present application. Accordingly, this specification and the drawings are merely exemplary illustrations of the present application as defined by the appended claims, and are deemed to have covered any and all modifications, variations, combinations or equivalents within the scope of the present application. Obviously, those skilled in the art may make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A data processing method, characterized in that: Applied to a data processing system, the data processing system includes an external system, an adaptation center and an internal system, including: In response to a data processing request from an external system, a data source that meets a preset specification is synchronized from the internal system through the adaptation center, and the data source is used by the adaptation center to determine the business scenario corresponding to the data processing request; Clean the data source according to the business scenario to obtain a target data source, and send the target data source to the external system; Wherein, synchronizing the data source that meets the preset specification from the internal system through the adaptation center includes: determining the data field carried in the data processing request of the external system; determining the data format corresponding to the data field; obtaining the preset data content corresponding to the data format from the internal system through the adaptation center; constructing a first dictionary space according to the preset data content; processing the first dictionary space to obtain multiple sub-first dictionary spaces, and the data sources corresponding to the multiple sub-first dictionary spaces are the data sources that meet the preset specification; Wherein, performing data cleaning on the data source according to the business scenario to obtain the target data source includes: constructing a second dictionary space according to the data field; calculating the distance between the second dictionary space and each of the multiple sub-first dictionary spaces to obtain multiple space distances; selecting multiple target space distances less than or equal to a preset distance from the multiple space distances; determining the target sub-first dictionary space corresponding to each of the target space distances to obtain multiple target sub-first dictionary spaces; determining the target data source according to the business scenario and the multiple target sub-first dictionary spaces; Among them, determining the target data source according to the business scenario and the multiple target sub-first dictionary spaces includes: determining a first privacy coefficient corresponding to the business scenario; determining the data source to be cleaned corresponding to the multiple target sub-first dictionary spaces; determining a second privacy coefficient corresponding to the external system according to a preset mapping relationship between the external system and the privacy coefficient; optimizing the first privacy coefficient by the second privacy coefficient to obtain a target privacy coefficient; determining multiple groups of data related to the business scenario from the data source to be cleaned; generating a data matrix corresponding to each group of data according to the multiple groups of data; generating a privacy matrix according to the target privacy coefficient; performing data cleaning on each data matrix according to the privacy matrix to obtain target data corresponding to each group of data, and the multiple target data constitute the target data source.

2. The method according to claim 1, characterized in that Before synchronizing the data source that meets the preset specification from the internal system through the adaptation center, the method further includes: Determining a data interface corresponding to the data processing request; Determining a target interface type of the data interface; Determine the target data synchronization mode corresponding to the target interface type according to the mapping relationship between the preset interface type and the data synchronization mode; According to the target data synchronization mode, the step of synchronizing the data source that meets the preset specification from the internal system through the adaptation center is performed.

3. The method according to claim 2, characterized in that The target data synchronization method includes any one of the following: interface synchronization, scheduled task synchronization and message queue synchronization.

4. A data processing device, characterized in that: Applied to a data processing system, the data processing system includes an external system, an adaptation center and an internal system, the device includes: a synchronization unit and a data cleaning unit, wherein: The synchronization unit is used to respond to a data processing request from an external system and synchronize a data source that meets a preset specification from the internal system through the adaptation center, wherein the data source is used by the adaptation center to determine a business scenario corresponding to the data processing request; The data cleaning unit is used to clean the data source according to the business scenario to obtain a target data source, and send the target data source to the external system; Wherein, synchronizing the data source that meets the preset specification from the internal system through the adaptation center includes: determining the data field carried in the data processing request of the external system; determining the data format corresponding to the data field; obtaining the preset data content corresponding to the data format from the internal system through the adaptation center; constructing a first dictionary space according to the preset data content; processing the first dictionary space to obtain multiple sub-first dictionary spaces, and the data sources corresponding to the multiple sub-first dictionary spaces are the data sources that meet the preset specification; Wherein, performing data cleaning on the data source according to the business scenario to obtain the target data source includes: constructing a second dictionary space according to the data field; calculating the distance between the second dictionary space and each of the multiple sub-first dictionary spaces to obtain multiple space distances; selecting multiple target space distances less than or equal to a preset distance from the multiple space distances; determining the target sub-first dictionary space corresponding to each of the target space distances to obtain multiple target sub-first dictionary spaces; determining the target data source according to the business scenario and the multiple target sub-first dictionary spaces; Among them, determining the target data source according to the business scenario and the multiple target sub-first dictionary spaces includes: determining a first privacy coefficient corresponding to the business scenario; determining the data source to be cleaned corresponding to the multiple target sub-first dictionary spaces; determining a second privacy coefficient corresponding to the external system according to a preset mapping relationship between the external system and the privacy coefficient; optimizing the first privacy coefficient by the second privacy coefficient to obtain a target privacy coefficient; determining multiple groups of data related to the business scenario from the data source to be cleaned; generating a data matrix corresponding to each group of data according to the multiple groups of data; generating a privacy matrix according to the target privacy coefficient; performing data cleaning on each data matrix according to the privacy matrix to obtain target data corresponding to each group of data, and the multiple target data constitute the target data source.

5. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory is used to store one or more programs and is configured to be executed by the processor, wherein the program comprises instructions for executing the steps in the method according to any one of claims 1 to 3.

6. A computer program product, wherein: The computer program product comprises a non-transitory computer-readable storage medium storing a computer program, wherein the computer program is operable to cause a computer to execute the method as described in any one of claims 1-3.

7. A computer-readable storage medium, characterized in that: A computer program for electronic data exchange is stored, wherein the computer program enables a computer to execute the method according to any one of claims 1 to 3.

Images