Multi-source data processing method, system, device and storage medium

Abstract

The application provides a multi-source data processing method, system, device and storage medium. The method comprises the following steps: receiving an added data request, and obtaining a primary key of the added data; when stored data with the same primary key as the added data is found, comparing the core field mark value and the data source of the added data and the stored data; when the core field mark value of the added data and the stored data is the same and the data sources are different, obtaining a first credibility score of the added data and a second credibility score of the stored data; and determining a data processing mode according to the comparison result of the first credibility score and the second credibility score, wherein the data processing mode comprises that the added data covers the stored data and the added data does not cover the stored data. By using the application, the safety and accuracy of mutual covering between data from different data sources are ensured.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a multi-source data processing method, system, device and storage medium. Background Technology

[0002] For a multi-source data management system, it interacts with various data sources to acquire, integrate, and store multi-source data. This multi-source data may originate from multiple data sources at the same level or from multiple data sources at different levels. Taking a field command system as an example, there may be situations where a command system is deployed at multiple levels, such as "county-city-province-ministerial" or "city-province-ministerial." Each level of the command system generates business data or simultaneously interfaces with multiple external business systems. It may also edit and modify data within the command system. Furthermore, each level of the command system needs to progressively aggregate updated business data to the next higher level. Multi-level, multi-source data integration may encounter the following problems:

[0003] (1) Data modifications made by higher-level users in the command system may be overwritten by data transmitted from lower-level command systems.

[0004] (2) When the superior command system reconnects with the data of an external business system at the same level, the revised data brought up from the subordinate level is overwritten by the historical data from the external business system.

[0005] (3) For the same data, the command systems at the upper and lower levels have connected to different external business systems at the same level. For example, they have connected to different vehicle management systems at the city and provincial levels, resulting in different unique primary key rules for the data. After the city-level data is connected to the provincial level, some data with the same primary key are overwritten.

[0006] (4) The primary key of two lower-level data sources is duplicated, which causes the data to be overwritten when it is transmitted to the higher level. For example, for personnel information, Li XX in City A has a primary key of 20X in the City A system, and Wang XX in City B also has a primary key of 20X in the City B system. When these two data are transmitted to the provincial command system at the same time, they will be overwritten by each other.

[0007] In summary, existing multi-source data management systems, when integrating data from multiple sources, cannot simultaneously ensure efficient data flow between these sources while guaranteeing the security and accuracy of data coverage between different sources. Summary of the Invention

[0008] In view of the problems in the prior art, the purpose of this application is to provide a multi-source data processing method, system, device and storage medium to ensure the security and accuracy of data overlapping between different data sources.

[0009] This application provides a multi-source data processing method, including the following steps:

[0010] Receive new data and retrieve the primary key of the new data;

[0011] When stored data with the same primary key as the newly added data is found, the core field marker values ​​and data sources of the newly added data and the stored data are compared.

[0012] When the core field tag values ​​of the newly added data and the stored data are the same but the data sources are different, obtain the first credibility score of the newly added data and the second credibility score of the stored data;

[0013] Based on the comparison results of the first credibility score and the second credibility score, a data processing method is determined, wherein the data processing method includes the new data overwriting the stored data and the new data not overwriting the stored data.

[0014] In the multi-source data processing method of this application, when new data is needed, stored data with the same primary key as the new data is queried. To avoid data overwriting errors, the new data is not directly overwritten with the stored data. Instead, the core field marker values ​​and data sources of the new data and the stored data are further compared. When the core field marker values ​​of the new data and the stored data are the same but the data sources are different, the reliability scores of the two are used to determine whether to use the new data to overwrite the stored data. This reduces the risk of data overwriting errors and ensures the security and accuracy of data overwriting between different data sources.

[0015] In some embodiments, comparing the core field marker values ​​and data sources of the newly added data and the stored data includes the following steps:

[0016] Obtain the values ​​of the core fields of the newly added data, and calculate the core field marker values ​​of the newly added data using a preset mapping algorithm;

[0017] Obtain the core field marker value of the stored data;

[0018] Determine whether the core field marker values ​​of the newly added data and the stored data are the same;

[0019] If the core field marker values ​​are the same, then determine whether the data sources of the newly added data and the stored data are the same;

[0020] If the core field marker values ​​are the same and the data sources are the same, then the data processing method is determined to be that the newly added data overwrites the already stored data;

[0021] If the core field tag values ​​are the same but the data sources are different, then obtain the first credibility score of the data source of the newly added data and the second credibility score of the stored data.

[0022] In some embodiments, after determining whether the core field marker values ​​of the newly added data and the stored data are the same, the following steps are also included:

[0023] If the core field marker values ​​are different, then determine whether the data sources of the newly added data and the stored data are the same;

[0024] If the core field flag values ​​are different and the data sources are the same, then the abnormal core field that has changed is identified;

[0025] An exception overwrite prompt is generated based on the core exception fields and pushed to the user's device;

[0026] Receive a data processing instruction from the user terminal. If the data processing instruction is a confirmation to overwrite, then determine that the data processing method is for the new data to overwrite the stored data.

[0027] If the core field marker values ​​are different and the data sources are different, then the data processing method is determined to be that the newly added data does not overwrite the already stored data and a data processing log is generated.

[0028] In some embodiments, obtaining a first credibility score for the newly added data and a second credibility score for the stored data includes the following steps:

[0029] Obtain the first data source of the newly added data and the second data source of the already stored data, respectively;

[0030] In the preset credibility evaluation strategy table, the first evaluation strategy that the newly added data conforms to is determined according to the first data source, and the second evaluation strategy that the stored data conforms to is determined according to the second data source. The first credibility score corresponding to the first evaluation strategy and the second credibility score corresponding to the second evaluation strategy are queried respectively.

[0031] In some embodiments, obtaining a first credibility score for the newly added data and a second credibility score for the stored data includes the following steps:

[0032] Obtain the data object type of the newly added data;

[0033] Obtain the first data source of the newly added data and the second data source of the already stored data, respectively;

[0034] Compare the values ​​of each field in the newly added data and the stored data to determine the fields involved in the change;

[0035] In the preset credibility evaluation strategy table, the first evaluation strategy that the newly added data conforms to is determined according to the data object type, the first data source and the relevant fields. The second evaluation strategy that the second data source conforms to is determined according to the data object type, the second data source and the relevant fields. The first credibility score corresponding to the first evaluation strategy and the second credibility score corresponding to the second evaluation strategy are queried respectively.

[0036] In some embodiments, the data processing method is determined based on the comparison result of the first confidence score and the second confidence score, including the following steps:

[0037] If the first credibility score is higher than the second credibility score, the data processing method is to overwrite the stored data with the new data, generate a data processing log, and push it to the user terminal.

[0038] If the first credibility score is equal to or lower than the second credibility score, the data processing method is that the new data does not overwrite the stored data, a data processing log is generated, and it is pushed to the user terminal.

[0039] In some embodiments, the following steps are also included:

[0040] Receive the modification instruction for the data processing method sent by the user terminal;

[0041] The data processing method is modified according to the modification instruction, and the credibility score corresponding to the first evaluation strategy and / or the credibility score corresponding to the second evaluation strategy in the credibility evaluation strategy table are modified.

[0042] This application embodiment also provides a multi-source data processing system applied to the aforementioned multi-source data processing method, the system comprising:

[0043] The data receiving module is used to receive new data and obtain the primary key of the new data;

[0044] The data comparison module is used to compare the core field marker values ​​and data sources of the newly added data and the stored data when the stored data with the same primary key as the newly added data is found.

[0045] The credibility evaluation module is used to obtain a first credibility score for the newly added data and a second credibility score for the stored data when the core field tag values ​​of the newly added data and the stored data are the same but the data sources are different.

[0046] The data processing module is used to determine the data processing method based on the comparison result of the first credibility score and the second credibility score. The data processing method includes the new data overwriting the stored data and the new data not overwriting the stored data.

[0047] In the multi-source data processing system of this application, by employing a data receiving module and a data comparison module, when new data is needed, stored data with the same primary key as the new data is queried. To avoid data erroneous overwriting, the new data is not directly overwritten with the stored data. Instead, the data comparison module further compares the core field marker values ​​and data sources of the new data and the stored data. Through the credibility evaluation module and the data processing module, when the core field marker values ​​of the new data and the stored data are the same but the data sources are different, the credibility scores of the two determine whether to use the new data to overwrite the stored data. This reduces the risk of data erroneous overwriting and thus ensures the security and accuracy of data overwriting between different data sources.

[0048] This application also provides a multi-source data processing device, including:

[0049] processor;

[0050] A memory in which executable instructions of the processor are stored;

[0051] The processor is configured to perform the steps of the multi-source data processing method by executing the executable instructions.

[0052] By employing the multi-source data processing device provided in this application, the processor executes the multi-source data processing method when executing the executable instructions, thereby obtaining the beneficial effects of the aforementioned multi-source data processing method.

[0053] This application also provides a computer-readable storage medium for storing a program, which, when executed by a processor, implements the steps of the multi-source data processing method.

[0054] By employing the computer-readable storage medium provided in this application, the program stored therein implements the steps of the multi-source data processing method when executed, thereby obtaining the beneficial effects of the multi-source data processing method described above. Attached Figure Description

[0055] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings.

[0056] Figure 1 This is a flowchart of a multi-source data processing method according to an embodiment of this application;

[0057] Figure 2 This is a schematic diagram illustrating the hierarchical deployment of multiple data processing systems according to an embodiment of this application;

[0058] Figure 3 This is a flowchart illustrating an embodiment of the present application, which compares the core field marker values ​​and data sources of the newly added data and the stored data, and determines the data processing method based on the comparison results.

[0059] Figure 4 This is a flowchart illustrating the process of obtaining a credibility score according to an embodiment of this application;

[0060] Figure 5 This is a schematic diagram of a multi-source data processing system according to an embodiment of this application;

[0061] Figure 6 This is a schematic diagram of the structure of a multi-source data processing device according to an embodiment of this application;

[0062] Figure 7 This is a schematic diagram of the structure of a computer storage medium according to an embodiment of this application. Detailed Implementation

[0063] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this application will be thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar structures, and therefore repeated descriptions of them will be omitted. Although the terms "first" or "second," etc., are used in this specification to denote certain features, these are merely indicative of function and not as a limitation on the number or importance of specific features.

[0064] like Figure 1 As shown, in one embodiment, this application provides a multi-source data processing method, including the following steps:

[0065] S100: Receive new data and obtain the primary key of the new data;

[0066] This multi-source data processing method can be executed by a data processing system whose database stores multiple sets of data. This data processing system can interact with data processing systems at other levels to obtain new data from other data processing systems or from business systems. Alternatively, staff can directly operate the data processing system to modify or add data at this level. Any addition or modification of data that results in the generation of new data is considered as generating new data.

[0067] Here, a primary key refers to one or more fields in a database table. Its value is used to uniquely identify a piece of data in the table. However, when there are multiple databases in multiple data processing systems, the primary key cannot achieve the purpose of uniquely identifying a piece of data across multiple databases. Therefore, if the database of the data processing system already stores stored data with the same primary key as the new data, directly overwriting it may lead to data loss, data errors, and other problems due to incorrect overwriting.

[0068] Figure 2 The example illustrates the structure of interaction between multiple data processing systems. Figure 2 The illustrated architecture deploys multiple levels, each capable of hosting one or more data processing systems. Each data processing system interfaces with one or more business systems, receiving new data from them. Lower-level data processing systems communicate with higher-level systems, meaning a data processing system can receive new data from its subordinate systems. Figure 2 For example, the diagram illustrates four levels, with the Level 1 data processing system being the central data processing system, the highest level, and the Level 4 data processing system the lowest level. The diagram exemplarily shows two Level 2 data processing systems A and B, two Level 3 data processing systems C and D, and three Level 4 data processing systems E, F, and H, but this is not intended to limit the scope of this application. For Level 2 data processing system A, the sources of its newly added data may include business systems M and N, lower-level Level 3 data processing systems C and D, and editing or corrections made by its own staff. For Level 3 data processing system C, the sources of its newly added data may include business systems P and O, Level 4 data processing systems E, F, and H, and editing or corrections made by its own staff. For Level 4 data processing system E, the sources of its newly added data may include business systems Q, R, and S, and editing or corrections made by its own staff. Figure 2The illustrated architecture can be applied to various scenarios. For example, in a command system scenario, the data processing system is categorized by administrative division: Level 1 is the central data processing system, Level 2 is the provincial system, Level 3 is the municipal system, and Level 4 is the county system. Business systems refer to third-party systems connected to the data processing system, such as those of fire safety companies or monitoring providers. If each data processing system relies solely on primary keys to determine data duplication, directly replacing data from different sources using the same primary key will result in erroneous data overwriting.

[0069] S200: When a stored data with the same primary key as the newly added data is found, the core field marker values ​​and data sources of the newly added data and the stored data are compared. Here, the data source refers to the relevant information of the data processing system that generated the newly added data.

[0070] S300: When the core field tag values ​​of the newly added data and the stored data are the same but the data sources are different, obtain the first credibility score of the newly added data and the second credibility score of the stored data;

[0071] The newly added data and the stored data mentioned here have the same core field identifier value but different data sources. This means that the newly added data and the stored data are generated from different data sources for the same core field identifier value. Therefore, it is necessary to determine the data to be retained based on the credibility scores of the two. Here, the core field identifier value is the identifier value that uniquely identifies the data.

[0072] S400: Based on the comparison result of the first credibility score and the second credibility score, determine the data processing method, wherein the data processing method includes the new data overwriting the stored data and the new data not overwriting the stored data;

[0073] Specifically, such as Figure 3 As shown, step S400 includes: when the first credibility score is higher than the second credibility score, it indicates that the credibility of the newly added data is higher, and proceed to S410: determine that the data processing method is for the newly added data to overwrite the stored data, and perform this overwriting action; when the first credibility score is equal to or lower than the second credibility score, it indicates that the credibility of the newly added data is lower or the same as the credibility of the stored data, and proceed to step S420: determine that the data processing method is for the newly added data not to overwrite the stored data, that is, not to perform the overwriting action, thereby avoiding the mistaken overwriting of data with higher credibility with data with lower credibility.

[0074] In the multi-source data processing method of this application, by employing steps S100 and S200, when new data is needed, stored data with the same primary key as the new data is queried. To avoid data erroneous overwriting, the new data is not directly overwritten by the stored data. Instead, step S200 further compares the core field marker values ​​and data sources of the new data and the stored data. Through steps S300 and S400, when the core field marker values ​​of the new data and the stored data are the same but the data sources are different, the reliability scores of the two are used to determine whether to use the new data to overwrite the stored data. This reduces the risk of data erroneous overwriting and thus ensures the security and accuracy of data overwriting between different data sources.

[0075] In this application, different data processing systems communicate and monitor data transfers and changes through the Kafka message middleware. For example, when a lower-level data processing system adds new data to the Kafka message middleware, the upper-level data processing system, upon detecting the new data, executes steps S100 to S400 to determine whether to overwrite existing locally stored data with the new data.

[0076] In this embodiment, with Figure 2 Taking the architecture as an example, for a data processing system, the new data it receives may come from multiple sources, leading to overlap between different data sources. Therefore, before executing steps S200-S400, the system can pre-screen the new data based on its source, eliminating obviously unsuitable data in advance. This avoids the need for further judgment in subsequent steps, thus saving system processing resources and improving data processing efficiency. Figure 2 For example, business system M may simultaneously interface with secondary data processing systems A and B. When transmitting data to systems A and B, it might transmit data from both systems A and B to system A. Therefore, when system A establishes data transfer with the next higher level, it might transfer data from both systems A and B to that higher level. When a primary data processing system receives new data from secondary data processing system A, this new data might include data from secondary data processing system A itself, as well as data from secondary data processing system B. However, data from system B is not within system A's data transfer permissions; therefore, the primary data processing system can directly discard the data from system B received from system A.

[0077] Specifically, the multi-source data processing system stores the data transmission permission range corresponding to each data source. Step S100, after receiving a new data request and obtaining the primary key of the new data, further includes: determining the data source of the new data and judging whether the new data falls within the data transmission permission range of its data source. If yes, proceed to steps S200-S400; if no, i.e., the new data is not within the data transmission permission range of this data source, discard the new data and end the current process. This data permission range can, for example, be the range of the administrative division corresponding to the data. When secondary data processing systems A and B correspond to the data processing systems of province A and province B respectively, when secondary data processing system A obtains data from the business system, it may obtain data from province A or province B. However, the data from province B is not within the data transmission permission range of system A, and system B will also transmit this data from province B to the primary data processing system, resulting in duplicate data transmission. The primary data processing system can directly discard the data from province B from system A, thereby saving resources and time for processing new data.

[0078] like Figure 3 As shown, in this embodiment, after receiving the new data and obtaining the primary key of the new data, step S100 further includes the following steps:

[0079] S101: Determine whether there is any existing data with the same primary key as the newly added data;

[0080] If yes, continue to step S200; if no, there is no need to consider whether to overwrite, and proceed to step S500: store the newly added data, that is, store the newly added data as a new piece of data locally.

[0081] like Figure 3 As shown, in this embodiment, step S200, comparing the core field marker values ​​and data sources of the newly added data and the stored data, includes the following steps:

[0082] S201: Obtain the value of the core field of the newly added data, and calculate the core field marker value of the newly added data using a preset mapping algorithm;

[0083] In this embodiment, the values ​​of multiple core fields of the newly added data are concatenated, and the concatenated values ​​are calculated using a preset mapping algorithm, such as MD5 algorithm or hash value calculation algorithm, to obtain the core field mark value of the newly added data. This core field mark value can uniquely identify the newly added data. For example, for personnel data, the personnel ID number + personnel name + personnel mobile phone number are used as core fields and concatenated.

[0084] In this embodiment, the primary key query and the core field query mentioned above can be cached using Redis to improve query efficiency;

[0085] S202: Obtain the core field marker value of the stored data;

[0086] The core field marker value can be pre-stored in the local database of the data processing system, or it can be calculated using a method similar to the one described above, that is, concatenating the values ​​of the core fields of the stored data and using a preset mapping algorithm to calculate the core field marker value.

[0087] The execution order of steps S202 and S201 is only an example. You can also execute step S202 first, then step S201, or execute steps S201 and S202 at the same time, etc.

[0088] S203: Determine whether the core field marker values ​​of the newly added data and the stored data are the same;

[0089] If the core field marker values ​​are the same, it means that the core field values ​​have not changed, and continue to S204: determine whether the data sources of the newly added data and the stored data are the same;

[0090] If the core field marker values ​​are the same and the data sources are the same, it means that the new data is an update of the stored data from the same data source, so it can be safely overwritten. Continue to S410: Determine that the data processing method is for the new data to overwrite the stored data, and perform this overwriting action.

[0091] If the core field marker values ​​are the same but the data sources are different, it means that the new data modifies the stored data from different data sources. It is necessary to further determine whether it can be covered based on the credibility score, and continue to step S300.

[0092] In this embodiment, if the core field marker values ​​are the same but the data sources are different, it indicates that different data sources have modified the same data. Before proceeding to step S300, the following steps are also included:

[0093] Obtain the first modification time recorded in the corresponding first data source for the newly added data, and the second modification time recorded in the corresponding second data source for the stored data;

[0094] If the first modification time is earlier than the second modification time, and the difference between the first modification time and the second modification time is greater than a preset time threshold, then continue with step S420 to avoid newly added data with a very early modification time from incorrectly overwriting stored data with a later modification time.

[0095] like Figure 3 As shown, in this embodiment, after step S203: determining whether the core field marker values ​​of the newly added data and the stored data are the same, the following steps are also included:

[0096] If the core field marker values ​​are different, it indicates that the value of at least one core field has changed, and proceed to S205: determine whether the data sources of the newly added data and the stored data are the same;

[0097] If the core field marker values ​​are different and the data sources are the same, then continue to S206: determine the abnormal core field that has changed, generate an abnormal overwrite prompt based on the abnormal core field, and push it to the user terminal; specifically, compare the values ​​of each core field of the newly added data with the values ​​of the core fields of the stored data to determine the abnormal core field that has changed. For example, if the comparison finds that the ID number of the newly added data is different from the ID number of the stored data, then the ID number is determined to be an abnormal core field, the abnormal core field is added to the abnormal overwrite prompt, and the abnormal overwrite prompt is pushed to the user terminal. The user can use the user terminal to view the abnormal overwrite prompt and the abnormal core field therein. Here, the user can be the administrator or staff of the data processing system.

[0098] S207: Receive data processing instructions from the user terminal; if the data processing instructions are confirmation of overwrite, then continue to step S410: determine the data processing method as the new data overwrites the stored data, and execute this overwrite action;

[0099] If the core field marker values ​​are different and the data sources are different, then proceed to step S420: determine that the data processing method is that the newly added data does not overwrite the already stored data.

[0100] like Figure 3 As shown, after steps S410 and S420, step S430 is further included: generating a data processing log and pushing it to the user terminal. The pushing of the data processing log in step S430 can be done by pushing the data processing log to the user terminal every time new data is processed, or by pushing the data processing log of new data added during a certain period of time, or by pushing it to the user terminal when the number of data processing log entries reaches a certain amount, etc. The data processing log may include the data processing method of the new data, the data type of the new data, the data source of the new data, and the reason why the new data did not overwrite the already stored data, if so.

[0101] In this embodiment, step S300, obtaining the first credibility score of the newly added data and the second credibility score of the stored data, includes the following steps:

[0102] The first data source of the newly added data and the second data source of the stored data are obtained respectively; here, the second data source of the stored data refers to the data source corresponding to the data source identifier added to the stored data when it was generated.

[0103] In this embodiment, when a data processing system generates a new piece of data, a data source identifier is added to the data to identify its corresponding data source;

[0104] In the preset credibility evaluation strategy table, the first evaluation strategy that the newly added data conforms to is determined according to the first data source, and the second evaluation strategy that the stored data conforms to is determined according to the second data source;

[0105] Query the first credibility score corresponding to the first evaluation strategy and the second credibility score corresponding to the second evaluation strategy from the credibility evaluation strategy table respectively;

[0106] The credibility evaluation strategy table can store evaluation strategy identifiers corresponding to different data sources. Each evaluation strategy identifier corresponds to a credibility score. Here, each evaluation strategy can directly include a credibility score value, or it can include a credibility score calculation rule. To determine the first evaluation strategy and the second evaluation strategy, the identifiers of the first evaluation strategy and the second evaluation strategy can be queried, and then the first credibility score and the second credibility score can be queried based on the identifiers of the first evaluation strategy and the second evaluation strategy.

[0107] In this embodiment, the evaluation strategies in the credibility evaluation strategy table can be further subdivided. Each evaluation strategy stored in the credibility evaluation strategy table corresponds to a data source type, data source information, and related fields. Therefore, the credibility evaluation strategy table is a multi-dimensional mapping table, with dimensions including, for example, data source type, data source information, and related fields. Based on a specific data source type, a specific data source information, and one or more specific related fields, an evaluation strategy identifier can be queried, and then the credibility score corresponding to that evaluation strategy identifier can be retrieved. In application, if an evaluation strategy identifier corresponding to a specific data source type, a specific data source information, and one or more specific related fields cannot be found, it is determined whether a universal evaluation strategy applicable to various related fields exists for that specific data source type and specific data source information. If such a universal evaluation strategy exists, it is adopted; if it does not exist, step S410 can be directly continued.

[0108] Specifically, such as Figure 4As shown, step S300, obtaining the first credibility score of the newly added data and the second credibility score of the stored data, includes the following steps:

[0109] S310: Obtain the data object type of the newly added data;

[0110] Data object type refers to the type of data that is newly added. For example, data object types can be pre-classified as fire hydrant data objects, monitoring data objects, traffic light data objects, etc.

[0111] S320: Obtain the first data source of the newly added data and the second data source of the stored data respectively;

[0112] In this embodiment, the data may come from different levels of data sources (such as provincial, municipal, and district levels, where the data source level is the vertical dimension), different types of data sources (such as the business systems of fire-related departments and transportation-related departments, where the data source type is the horizontal dimension), and different administrative division data sources (such as a specific province, city, or district). Obtaining the first data source includes obtaining the data source level, data source type, and administrative division of the first data source for newly added data. Obtaining the second data source includes obtaining the data source level, data source type, and administrative division of the stored data.

[0113] S330: Compare the values ​​of each field in the newly added data and the stored data to determine the fields involved in the change; the fields involved here are the fields whose values ​​are different between the newly added data and the stored data.

[0114] S340: In the preset credibility evaluation strategy table, determine the first evaluation strategy that the newly added data conforms to based on the data object type, the first data source and the relevant fields, and determine the second evaluation strategy that the second data source conforms to based on the data object type, the second data source and the relevant fields;

[0115] S350: Query the first credibility score corresponding to the first evaluation strategy and the second credibility score corresponding to the second evaluation strategy from the credibility evaluation strategy table.

[0116] In this embodiment, a first evaluation strategy identifier is retrieved based on the data object type, the data source level of the first data source, the data source type and administrative division, and the relevant fields. A second evaluation strategy identifier is then retrieved based on the data object type, the data source level of the second data source, the data source type and administrative division, and the relevant fields. The corresponding first and second credibility scores are then determined based on the first and second evaluation strategy identifiers. The credibility evaluation strategy table can be pre-set manually by staff, taking into account factors such as the credibility of various types of data in historical data, customer feedback from different data processing / business systems, and the correlation between different data processing / business systems and various data object types. It can also be continuously optimized and adjusted based on new data processing methods during subsequent use. Figure 2 Taking the architecture as an example, the secondary data processing system A can obtain fire protection data of city C from the tertiary data processing system C, and can also obtain fire protection data of various cities in province A (including fire protection data of city C) from the business system M. For the fire protection data of city C, the credibility score of the tertiary data processing system C can be set to be higher than the credibility score of the business system M.

[0117] In this embodiment, after step S430: generating data processing logs and pushing them to the user terminal, the multi-source data processing method further includes the following steps:

[0118] Receive the modification instruction for the data processing method sent by the user terminal;

[0119] The data processing method is modified according to the modification instruction, and the credibility score corresponding to the first evaluation strategy and / or the credibility score corresponding to the second evaluation strategy in the credibility evaluation strategy table are modified.

[0120] For example, if the modification instruction is to change the previously covered data processing method to an uncovered data processing method, then the covered data needs to be changed back to its state before being covered, and the credibility score corresponding to the first evaluation strategy needs to be lowered, or the credibility score corresponding to the second evaluation strategy needs to be raised, so that the credibility score corresponding to the first evaluation strategy is lower than the credibility score corresponding to the second evaluation strategy. If the modification instruction is to change the previously uncovered data processing method to a covered data processing method, then the newly added data needs to overwrite the stored data, and the credibility score corresponding to the first evaluation strategy needs to be raised, or the credibility score corresponding to the second evaluation strategy needs to be lowered, so that the credibility score corresponding to the first evaluation strategy is higher than the credibility score corresponding to the second evaluation strategy.

[0121] like Figure 5 As shown in the illustration, this application also provides a multi-source data processing system applied to the aforementioned multi-source data processing method. The system includes:

[0122] The data receiving module M100 is used to receive new data and obtain the primary key of the new data;

[0123] The data comparison module M200 is used to compare the core field marker values ​​and data sources of the newly added data and the stored data when the stored data with the same primary key as the newly added data is found.

[0124] The credibility evaluation module M300 is used to obtain a first credibility score for the newly added data and a second credibility score for the stored data when the core field tag values ​​of the newly added data and the stored data are the same but the data sources are different.

[0125] The data processing module M400 is used to determine the data processing method based on the comparison result of the first credibility score and the second credibility score. The data processing method includes the new data overwriting the stored data and the new data not overwriting the stored data.

[0126] In the multi-source data processing system of this application, by employing a data receiving module M100 and a data comparison module M200, when new data is needed, stored data with the same primary key as the new data is retrieved. To avoid data erroneous overwriting, the new data is not directly overwritten by the stored data. Instead, the data comparison module M200 further compares the core field marker values ​​and data sources of the new data and the stored data. Furthermore, through the credibility evaluation module M300 and the data processing module M400, when the core field marker values ​​of the new data and the stored data are the same but their data sources are different, the credibility scores of both determine whether to use the new data to overwrite the stored data. This reduces the risk of data erroneous overwriting and thus ensures the security and accuracy of data overwriting between different data sources. The functions of each module in this multi-source data processing system can be implemented using the corresponding steps in the above-described multi-source data processing method.

[0127] This application also provides a multi-source data processing device, including a processor; a memory storing executable instructions of the processor; wherein the processor is configured to perform the steps of the multi-source data processing method by executing the executable instructions.

[0128] Those skilled in the art will understand that various aspects of this application can be implemented as a system, method, or program product. Therefore, various aspects of this application can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, collectively referred to herein as a "circuit," "module," or "system."

[0129] The following reference Figure 6 To describe an electronic device 600 according to this embodiment of the present application. Figure 6 The electronic device 600 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0130] like Figure 6 As shown, the electronic device 600 is presented in the form of a general-purpose computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different system components (including storage unit 620 and processing unit 610), a display unit 640, etc.

[0131] The storage unit stores program code that can be executed by the processing unit 610, causing the processing unit 610 to perform the steps described in the multi-source data processing method section of this specification according to various exemplary embodiments of this application. For example, the processing unit 610 can perform actions such as... Figure 1 The steps are shown in the figure.

[0132] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache storage unit 6202, and may further include a read-only memory unit (ROM) 6203.

[0133] The storage unit 620 may also include a program / utility 6204 having a set (at least one) program module 6205, such program module 6205 including but not limited to: an operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0134] Bus 630 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0135] Electronic device 600 can also communicate with one or more external devices 700 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 600, and / or with any device that enables electronic device 600 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 650. Furthermore, electronic device 600 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 660. Network adapter 660 can communicate with other modules of electronic device 600 via bus 630. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0136] By employing the multi-source data processing device provided in this application, the processor executes the multi-source data processing method when executing the executable instructions, thereby obtaining the beneficial effects of the aforementioned multi-source data processing method.

[0137] This application also provides a computer-readable storage medium for storing a program that, when executed by a processor, implements the steps of the multi-source data processing method. In some possible implementations, various aspects of this application can also be implemented as a program product comprising program code that, when run on a terminal device, causes the terminal device to perform the steps described in the multi-source data processing method section of this specification according to various exemplary embodiments of this application.

[0138] refer to Figure 7 As shown, a program product 800 for implementing the above-described method according to an embodiment of this application is described. It may employ a portable compact disc read-only memory (CD-ROM) and include program code, and can run on a terminal device, such as a personal computer. However, the program product of this application is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0139] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0140] The computer-readable storage medium may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, capable of transmitting, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0141] Program code for performing the operations of this application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or cluster. In cases involving remote computing devices, the remote computing devices can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to external computing devices (e.g., via the Internet using an Internet service provider).

[0142] By employing the computer-readable storage medium provided in this application, the program stored therein implements the steps of the multi-source data processing method when executed, thereby obtaining the beneficial effects of the multi-source data processing method described above.

[0143] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of this application and should not be construed as limiting the specific implementation of this application to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of this application, and all such modifications or substitutions should be considered within the scope of protection of this application.

Claims

1. A multi-source data processing method, characterized in that, Includes the following steps: Receive new data and retrieve the primary key of the new data; When stored data with the same primary key as the newly added data is found, the core field marker value and data source of the newly added data and the stored data are compared. The core field marker value is calculated based on the value of the core field using a preset mapping algorithm. When the core field marker values ​​of the newly added data and the stored data are different but the data sources are the same, the abnormal core field that has changed is determined. An exception overwrite prompt is generated based on the core exception fields and pushed to the user's device; Receive a data processing instruction from the user terminal. If the data processing instruction is a confirmation to overwrite, then determine that the data processing method is for the new data to overwrite the stored data. When the core field tag values ​​of the newly added data and the stored data are the same but the data sources are different, obtain the first credibility score of the newly added data and the second credibility score of the stored data; Based on the comparison results of the first credibility score and the second credibility score, a data processing method is determined, wherein the data processing method includes the new data overwriting the stored data and the new data not overwriting the stored data; The process of obtaining the first credibility score of the newly added data and the second credibility score of the stored data includes the following steps: Obtain the data object type of the newly added data; Obtain the first data source of the newly added data and the second data source of the already stored data, respectively; Compare the values ​​of each field in the newly added data and the stored data to determine the fields in which the values ​​of the newly added data and the stored data differ. In the preset credibility evaluation strategy table, the first evaluation strategy that the newly added data conforms to is determined according to the data object type, the first data source and the relevant fields. The second evaluation strategy that the second data source conforms to is determined according to the data object type, the second data source and the relevant fields. The first credibility score corresponding to the first evaluation strategy and the second credibility score corresponding to the second evaluation strategy are queried respectively.

2. The multi-source data processing method according to claim 1, characterized in that, The comparison of the core field marker values ​​and data sources of the newly added data and the stored data includes the following steps: Obtain the values ​​of the core fields of the newly added data, and calculate the core field marker values ​​of the newly added data using a preset mapping algorithm; Obtain the core field marker value of the stored data; Determine whether the core field marker values ​​of the newly added data and the stored data are the same; If the core field marker values ​​are the same, then determine whether the data sources of the newly added data and the stored data are the same; If the core field marker values ​​are the same and the data sources are the same, then the data processing method is determined to be that the newly added data overwrites the already stored data; If the core field tag values ​​are the same but the data sources are different, then obtain the first credibility score of the data source of the newly added data and the second credibility score of the stored data.

3. The multi-source data processing method according to claim 2, characterized in that, After determining whether the core field marker values ​​of the newly added data and the stored data are the same, the following steps are also included: If the core field marker values ​​are different and the data sources are different, then the data processing method is determined to be that the newly added data does not overwrite the already stored data and a data processing log is generated.

4. The multi-source data processing method according to claim 1, characterized in that, Based on the comparison results of the first credibility score and the second credibility score, the data processing method is determined, including the following steps: If the first credibility score is higher than the second credibility score, the data processing method is to overwrite the stored data with the new data, generate a data processing log, and push it to the user terminal. If the first credibility score is equal to or lower than the second credibility score, the data processing method is that the new data does not overwrite the stored data, a data processing log is generated, and it is pushed to the user terminal.

5. The multi-source data processing method according to claim 4, characterized in that, It also includes the following steps: Receive the modification instruction for the data processing method sent by the user terminal; The data processing method is modified according to the modification instruction, and the credibility score corresponding to the first evaluation strategy and / or the credibility score corresponding to the second evaluation strategy in the credibility evaluation strategy table are modified.

6. A multi-source data processing system, characterized in that, The system, applied to the multi-source data processing method according to any one of claims 1 to 5, comprises: The data receiving module is used to receive new data and obtain the primary key of the new data; The data comparison module is used to compare the core field marker values ​​and data sources of the newly added data and the stored data when the stored data with the same primary key as the newly added data is found. The credibility evaluation module is used to obtain a first credibility score for the newly added data and a second credibility score for the stored data when the core field tag values ​​of the newly added data and the stored data are the same but the data sources are different. The data processing module is used to determine the data processing method based on the comparison result of the first credibility score and the second credibility score. The data processing method includes the new data overwriting the stored data and the new data not overwriting the stored data.

7. A multi-source data processing device, characterized in that, include: processor; A memory in which executable instructions of the processor are stored; The processor is configured to perform the steps of the multi-source data processing method according to any one of claims 1 to 5 by executing the executable instructions.

8. A computer-readable storage medium for storing a program, characterized in that, When the program is executed by the processor, it implements the steps of the multi-source data processing method according to any one of claims 1 to 5.

Images