Data processing method, electronic device, storage medium and program product

Abstract

The invention provides a data processing method, electronic equipment, a storage medium and a program product. The data processing method comprises the steps of obtaining to-be-processed data; acquiring historical data related to the to-be-processed data; acquiring to-be-adopted data from the to-be-processed data and the historical data based on the adoption strategy rule; and writing the to-be-adopted data into the target database.

Description

Technical Field

[0001] The present disclosure relates to the technical field of computers, etc., and particularly relates to a data processing method, an electronic device, a storage medium, and a program product. Background Art

[0002] In the context of the rapid development of informatization and digitalization, data has become an important resource for promoting social and industrial development. In various application scenarios, the collection and storage of data are the basic links for realizing information processing and business analysis. By efficiently and accurately collecting data and storing and managing it in a reasonable manner, strong support can be provided for enterprise decision-making, system optimization, and technological innovation.

[0003] To ensure the timeliness and effectiveness of data, the Latest Data Rule, as a commonly used data adoption strategy, has been widely applied in multiple business scenarios. The Latest Data Rule ensures the leading position of the adopted data in terms of timeliness by preferentially adopting the data with the latest timestamp.

[0004] However, the adopted latest data may have errors or anomalies. If only the Latest Data Rule is used as the data adoption strategy, it may introduce incorrect or problematic data, thereby leading to a decline in data quality. Summary of the Invention

[0005] The present disclosure provides a data processing method, an electronic device, a storage medium, and a program product.

[0006] According to one aspect of the present disclosure, there is provided a data processing method, including: Obtaining data to be processed; Obtaining historical data related to the data to be processed; Obtaining data to be adopted from the data to be processed and the historical data based on an adoption strategy rule; Writing the data to be adopted into a target database.

[0007] According to the data processing method of at least one embodiment of the present disclosure, the obtaining of the historical data related to the data to be processed includes: Controlling to send a half-message carrying the data to be processed to a message middleware, and the status of the half-message is to be consumed; Writing the data to be processed into the target database; In response to the successful writing of the data to be processed, controlling to send a write success message to the message middleware; In response to the message middleware changing the status of the half-message to consumable according to the write success message, obtaining the historical data related to the data to be processed; Optionally, after the successful writing of the data to be processed, the method further includes: Generating a data acquisition record and a data version record of the data to be processed; Optionally, after writing the data to be processed into the target database, in response to the failure of writing the data to be processed, the method further includes: Controlling the target database to roll back and controlling the message middleware to roll back the half message.

[0008] According to the data processing method of at least one embodiment of the present disclosure, the step of obtaining the data to be processed includes: In response to receiving a data operation request, recording the data operation request; Obtaining the data to be processed to be operated according to the data operation request; Optionally, the step of recording the data operation request includes: Performing a permission check on the data operation request to obtain a check result; When the check result is a successful check, recording the data operation request.

[0009] According to the data processing method of at least one embodiment of the present disclosure, after writing the data to be adopted into the target database, the method further includes: Generating a data adoption record of the data to be adopted.

[0010] According to the data processing method of at least one embodiment of the present disclosure, the step of obtaining the historical data related to the data to be processed includes: Judging whether the data to be processed needs to be processed based on the adoption policy rule; If so, obtaining the historical data related to the data to be processed.

[0011] According to the data processing method of at least one embodiment of the present disclosure, if not, the method further includes: Writing the data to be processed into the target database as the data to be adopted.

[0012] According to the data processing method of at least one embodiment of the present disclosure, the step of obtaining the data to be adopted from the data to be processed and the historical data based on the adoption policy rule includes: Obtaining the data source scores of the data to be processed and the historical data; Obtaining target data from the data to be processed and the historical data according to the high and low of the data source scores; Judging whether the number of the target data is equal to 1; If it is not equal to 1, obtain the data to be adopted from the target data based on the adoption strategy rule; If it is equal to 1, use the target data as the data to be adopted.

[0013] According to another aspect of the present disclosure, there is provided an electronic device, including: a memory that stores execution instructions; and a processor that executes the execution instructions stored in the memory, so that the processor executes the data processing method according to any one of the embodiments of the present disclosure.

[0014] According to still another aspect of the present disclosure, there is provided a readable storage medium in which execution instructions are stored, and when the execution instructions are executed by a processor, they are used to implement the data processing method according to any one of the embodiments of the present disclosure.

[0015] According to yet another aspect of the present disclosure, there is provided a computer program product, including a computer program, and when the computer program is executed by a processor, it implements the data processing method according to any one of the embodiments of the present disclosure. Description of the Drawings

[0016] The drawings illustrate exemplary embodiments of the present disclosure and are used, together with the description thereof, to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are included in this specification and form a part of this specification.

[0017] Figure 1 It is a schematic diagram of an application scenario of the data processing method according to an embodiment of the present disclosure.

[0018] Figure 2 It is the flow of the data processing method according to an embodiment of the present disclosure Figure 1 .

[0019] Figure 3 It is Figure 2 the flow of the historical data acquisition method in the data processing method shown Figure 1 .

[0020] Figure 4 It is Figure 2 the flow of the historical data acquisition method in the data processing method shown Figure 2 .

[0021] Figure 5 It is Figure 2 the flow of the historical data acquisition method in the data processing method shown Figure 3 .

[0022] Figure 6 It is Figure 2 the flowchart of the method for obtaining data to be processed in the data processing method shown.

[0023] Figure 7 is Figure 6 The flowchart of the request recording method in the data acquisition method to be processed shown in the figure.

[0024] Figure 8 is the process of the data processing method according to an embodiment of the present disclosure Figure 2 .

[0025] Figure 9 is Figure 2 The flowchart of the historical data acquisition method in the data processing method shown in the figure. Figure 4 .

[0026] Figure 10 is Figure 2 The flowchart of the historical data acquisition method in the data processing method shown in the figure. Figure 5 .

[0027] Figure 11 is Figure 2 The flowchart of the data to be adopted acquisition method in the data processing method shown in the figure.

[0028] Figure 12 is the schematic flowchart of the data processing method according to the embodiment of the present disclosure.

[0029] Figure 13 is the schematic block diagram of the structure of the data processing device according to another embodiment of the present disclosure.

[0030] Figure 14 is the schematic block diagram of the structure of the electronic device according to still another embodiment of the present disclosure. Detailed implementation manners

[0031] The present disclosure will be further described in detail below with reference to the accompanying drawings and examples. It can be understood that the specific examples described herein are only used to explain the relevant content and do not limit the present disclosure. In addition, it should be noted that for the sake of description, only the parts related to the present disclosure are shown in the drawings.

[0032] It should be noted that, without conflict, the embodiments in the present disclosure and the features in the embodiments can be combined with each other. The technical solutions of the present disclosure will be described in detail below with reference to the drawings and in combination with the embodiments.

[0033] Taking the real estate dictionary as an example, the real estate dictionary is a structured data set used to store and manage information related to real estate projects. It is similar to an information repository or directory, aiming to systematically and standardize the sorting and recording of various real estate project information. It covers information in multiple dimensions and provides comprehensive and detailed real estate project information support for all aspects of the real estate industry, including but not limited to development, sales, leasing, evaluation, property management, etc. With the enrichment of business scenarios related to the real estate dictionary, the roles involved in underlying data production (such as real estate agents, auditors, etc.), channels (such as data collection tasks, error correction tasks, etc.) are also increasing day by day, and the quality of the produced data is uneven. Suppose the relevant information of a certain real estate project already exists in the real estate dictionary, and the current information includes: data source "provided by the developer", real estate project name "XX Century City", address "No. D, Street C, District B, City A", number of buildings "10 buildings", input time "January, 1st, 2025"; if an external user introduces new data for this information, including: data source "user feedback", real estate project name "XX Century City", address "No. D, Street C, District B, City A", number of buildings "8 buildings", input time "January, 15th, 2025". The system selects the data with a later timestamp according to the latest data rule, that is, the new data introduced by the external user; however, the new data introduced by the external user may be incorrect due to factors such as limited information sources, and the incorrect data will reduce the data quality, which will in turn lead to incorrect business decisions, such as miscalculating the total number of households in the real estate project, affecting decisions such as sales strategies and inventory management.

[0034] For this reason, the present disclosure proposes a data processing method, an electronic device, a readable storage medium, and a computer program product. The present disclosure can be implemented by data processing software installed on an electronic device such as a server.

[0035] Figure 1 Fig. shows an application scenario of the data processing method according to an embodiment of the present disclosure. In this application scenario, it may include a business system 100 and a data processing system 200, and the business system 100 is connected to the data processing system 200 through a network. The business system 100 is the source and producer of data, and its main function is to generate and manage data through the execution of business processes and provide this data to the data processing system 200; the data processing system 200 is the consumer and processor of data, and its main function is to receive data from the business system 100 and process the data to support downstream decision-making or business applications.

[0036] For the convenience of description and to make the technical solutions of the specific embodiments of the present disclosure easier to understand, before describing the data processing method implemented by the present disclosure, the technical terms involved in the specific embodiments of the present disclosure are explained as follows: The adoption strategy rule is a set of rule systems for selecting the optimal data from multiple data.

[0037] A semi-message is a message that is temporarily stored in the message middleware.

[0038] Message middleware is a software system used to transmit messages between different applications, system components, or services.

[0039] Rollback means undoing an operation and restoring the state before the operation.

[0040] Permission verification refers to the process of verifying and controlling the permissions of users or systems.

[0041] Data source scoring is a quantitative scoring result used to evaluate the performance of a data source.

[0042] Figure 2 FIG. 1 is a schematic diagram showing the overall flow of a data processing method M100 according to an embodiment of the present disclosure. Figure 2 The data processing method shown includes steps S110 to S140, wherein the data processing method can be executed by an electronic device such as a server.

[0043] Specifically, Figure 2 The data processing method shown includes: Step S110: Obtain data to be processed.

[0044] In some embodiments of the present disclosure, the data to be processed obtained through step S110 may involve multiple fields and different types of data; for example: the data to be processed obtained may specifically be transaction data, financial data, consumer behavior data and other data in the financial and economic fields; the data to be processed obtained may also be patient data, genomic data, environmental health data and other data in the medical and health fields, or student learning-related data, student behavior data, teaching content data and other data in the education field, or user behavior data, advertising data, market analysis data and other data in the business and marketing fields; the data to be processed obtained may also be device data, network log data, user experience data and other data in the technology and Internet fields.

[0045] It is understandable that before using the technical solutions disclosed in the various embodiments of this disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved in this disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0046] For example, in response to a user's active request, a prompt message is sent to the user to clearly inform the user that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the electronic device, application, server, storage medium, or other software or hardware that performs the operations of the disclosed technical solution based on the prompt message.

[0047] As an optional but non-limiting implementation, in response to receiving an active request from a user, the way of sending a prompt message to the user can be, for example, in the form of a pop-up window, and the prompt message can be presented in text in the pop-up window. In addition, the pop-up window can also carry selection controls for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0048] It can be understood that the above notification and the process of obtaining user authorization are only illustrative and do not constitute a limitation on the implementation of the present disclosure. Other ways that meet relevant laws and regulations can also be applied to the implementation of the present disclosure.

[0049] At the same time, it can be understood that the data involved in the technical solution of the present disclosure (including but not limited to the data itself, the acquisition or use of the data) should comply with the requirements of corresponding laws, regulations and related provisions.

[0050] Step S120: Obtain historical data related to the data to be processed.

[0051] In some embodiments of the present disclosure, the historical data is data stored within a period of time in the past (i.e., before performing a certain operation); the historical data related to the data to be processed obtained through step S120 can specifically be the field-level historical data of the data to be processed, that is, the historical data corresponding to the fields related to the data to be processed.

[0052] Step S130: Obtain the data to be adopted from the data to be processed and the historical data based on the adoption policy rule.

[0053] In some embodiments of the present disclosure, step S130 can obtain the optimal data to be adopted from the data to be processed and the historical data based on the adoption policy rule; the adoption policy rule can be set according to specific business scenarios and objectives (such as reliability, real-time performance, etc.). For example, the adoption policy rule can include: preferring data with high confidence, selecting data based on trends or patterns, selecting data based on coverage, obtaining the data to be adopted through modes, weighted averages or data fusion technologies, etc.

[0054] In the actual use process, the adoption policy rule can be adjusted according to real-time business objectives; dynamic weights or conditional rules can also be designed to support adjusting priorities in different business scenarios; a scoring model can also be introduced to optimize the adoption policy rule, improve the intelligence and consistency of data adoption, and achieve precise screening of high-quality data; an anomaly detection mechanism can also be introduced to filter invalid or abnormal data through the anomaly detection mechanism to ensure data quality and system stability.

[0055] Step S140: Write the data to be adopted into the target database.

[0056] In some embodiments of the present disclosure, the target database in step S140 may specifically be a standard library, which is used to uniformly manage and store the data processed through adopted policy rules and the like. Step S140 may write the data to be adopted into the target database through a standard library abstraction entity (an entity model that abstracts and standardizes core business data), thereby improving the data processing efficiency.

[0057] The data processing method provided by the present disclosure writes the optimal data to be adopted obtained from the data to be processed and historical data based on the adopted policy rules. It can make up for the deficiencies of the data to be processed through historical data and improve the quality and reliability of the data finally written into the database. This method solves the problem in the prior art that the latest data adopted may have errors or abnormal situations. If only the latest data rules are used as the data adoption strategy, it may introduce incorrect or problematic data, thereby leading to a decline in data quality.

[0058] In some embodiments of the present disclosure, steps S110 to S140 may be executed synchronously to improve the process control ability; steps S110 to S140 may also be executed asynchronously to improve the data processing efficiency.

[0059] After writing the data to be adopted into the target database through step S140, a data adoption notice and / or a data change notice may be sent to the business system related to the data to be processed and / or other systems that need to perceive the data writing; among them, the data adoption notice is a notice sent when the data to be adopted is determined after the above processing process, which is used to clarify which data is officially adopted as valid data after being screened by the adopted policy rules; the data change notice is a notice sent when the data stored in the target database changes, which is used to ensure that all systems or modules using this data can be updated in time to avoid errors or problems caused by data inconsistency.

[0060] Regarding step S120, in some embodiments of the present disclosure, it may include steps S121 to S124 as shown in Figure 3 ; Step S121, controlling to send a half-message carrying the data to be processed to the message middleware.

[0061] In some embodiments of the present disclosure, the status of the half-message in step S121 is to be consumed. After receiving the half-message, the message middleware stores it as a "temporary message" with the status of "to be consumed". The half-message will not be immediately consumed at this time, but waits for the relevant business logic to be executed and then submitted or rolled back.

[0062] The use of half-messages enables the entire data processing process to either succeed completely or fail completely, which can ensure data consistency.

[0063] In step S121, the message middleware can be in the form of a distributed message middleware, a message queue, etc., and the appropriate form of the message middleware can be selected in combination with the application scenario, performance requirements, complexity requirements, etc.

[0064] Step S122: Write the data to be processed into the target database.

[0065] In some embodiments of the present disclosure, the method of writing the data to be processed into the target database through step S122 is the same as the method of writing the data to be adopted into the target database through step S140.

[0066] Step S123: In response to the successful writing of the data to be processed, control to send a write success message to the message middleware.

[0067] In some embodiments of the present disclosure, when the message middleware receives the write success message, it will change the status of the half-message from "pending consumption" to "consumable" status, and the "consumable" status represents that the next business logic can be triggered.

[0068] Step S124: In response to the message middleware changing the status of the half-message to consumable according to the write success message, obtain the historical data related to the data to be processed.

[0069] Steps S121 to S124 decouple and asynchronously process the data acquisition and consumption processes through half-messages, improving the data processing efficiency; and the half-messages allow processing after the transaction is completed, avoiding being mis-consumed when the business logic is not completed, and improving data consistency.

[0070] Regarding step S120, in some embodiments of the present disclosure, after step S122, it may further include step S125 as Figure 4 shown below.

[0071] Step S125: Generate a data acquisition record and a data version record for the data to be processed.

[0072] In some embodiments of the present disclosure, the data acquisition record generated through step S125 is used to record the relevant information for acquiring the data to be processed; this data acquisition record generally includes: operation time (i.e., the specific time when the acquisition operation of the data to be processed occurred), operator or operating system (i.e., which user or which system triggered the acquisition operation of the data to be processed), data source (i.e., the place where the data to be processed was produced), and other information.

[0073] The data version generated through step S125 records the data corresponding to the fields to be processed. Each change to the data saves independent version information, which is used to record the specific state of the field at different times. This data version record generally includes: version number (uniquely identifying a certain version of the data of the field), data snapshot (recording the complete state of the data at this version), version time (used to mark the specific time when this version was created), reason for change (used to describe the reason why this version was created), operator or source (used to record who or which system produced the data of this version), etc.

[0074] When the data to be processed involves data corresponding to more than two fields, step S125 can generate data acquisition records and data version records separately for each field.

[0075] Taking the data to be processed as real estate-related data as an example, the data acquisition record can include: operation time, operation ID (a unique identifier used to identify different data acquisition operations), data source, user ID (a unique identifier used to identify the user or system that executes the data acquisition operation), city ID (a unique identifier used to identify the city related to the data), client key (a key used to identify the client used by the data source), evidence ID (a unique identifier used to identify the evidence (such as documents, files, records, etc.) related to the data acquisition), and additional additional information.

[0076] Taking the data to be processed as real estate-related data as an example, the data version record can include: primary key ID (a unique identifier used to identify the data to be processed), entity ID (a unique identifier used to identify the specific entity associated with the data version), entity type (used to describe the category to which the entity belongs), entity field (used to describe the name of a certain attribute or field in the entity), modified value (used to describe the specific data content corresponding to the entity), operation type (used to describe the operation type related to this data), operation ID, operation time, user ID, data source, span ID (a unique identifier used to identify the data processing span).

[0077] Step S125 records the data acquisition behavior through the data acquisition record, which is convenient for tracing the acquisition process and the root cause of problems; it records the changes of the data object itself through the data version record, which is convenient for data recovery and change comparison analysis.

[0078] Regarding step S120, in some embodiments of the present disclosure, after step S122, in response to the failure of writing the data to be processed, it may further include step S126 as Figure 5 shown.

[0079] Step S126 controls the target database to roll back and controls the message middleware to roll back the half message.

[0080] In some embodiments of the present disclosure, the failure to write the data to be processed means that the transaction is not completed, and the half-message cannot enter the "consumable" state. The target database can be rolled back through step S126, so that the write failure will not affect the target database; and the message middleware is controlled to roll back the half-message through step S126, that is, the half-message will be discarded and will not enter the consumer queue.

[0081] After rolling back the half-message through step S126, the message middleware will no longer retain the half-message, thereby releasing resources and avoiding the long-term occupation of memory or storage space by the half-message.

[0082] In particular, in order to improve the robustness and reliability of data processing, a transaction callback mechanism for half-messages can also be set; that is, when the message middleware has not confirmed the half-message for a long time, through the transaction callback mechanism, it asks about the status of the transaction corresponding to the half-message; if the status is successful, step S123 is executed; otherwise, step S124 is executed.

[0083] Regarding step S110, in some embodiments of the present disclosure, it may include steps S111 to S112 as Figure 6 shown.

[0084] Step S111, in response to receiving a data operation request, record the data operation request.

[0085] In some embodiments of the present disclosure, before formally processing the data operation request, the detailed information of the data operation request such as the request content, operation type, data to be processed, etc. can be recorded through step S111. Usually, the detailed information of the data operation request is recorded in a certain temporary table, memory structure or log, aiming to provide a complete basis for subsequent transaction processing and ensure the consistency and traceability of data processing.

[0086] Step S111 is mainly applicable to scenarios that need to efficiently process batch data operation requests; taking the data to be processed as real estate-related data as an example, the records of the data operation request may include: primary key ID, batch ID (a unique identifier for identifying a batch of data), operation ID, operation time, client key, modified value, execution status (used to describe whether the data has been processed), etc.

[0087] The operations corresponding to the data operation request may include insert, update, merge, batch insert, batch update, batch merge, etc.

[0088] Step S112, obtain the data to be processed that the data operation request is to operate according to the data operation request.

[0089] In some embodiments of the present disclosure, the specific process of obtaining the data to be processed according to the data operation request in step S112 may include: First, parse the data operation request to obtain the operation target and rules; then obtain the data to be processed according to the operation target and rules.

[0090] Through steps S111 to S112, it can be ensured that all data operation requests can be recorded and verified before actual processing, and it can also achieve a one-time submission of local transactions during batch processing.

[0091] Regarding step S111, in some embodiments of the present disclosure, it may include steps S1111 to S1112 as Figure 7 shown.

[0092] Step S1111, perform permission verification on the data operation request to obtain a verification result.

[0093] In some embodiments of the present disclosure, the permissions verified through step S1111 may be user-level permissions (i.e., whether the user's identity is true and valid), operation-level permissions (i.e., whether the user is allowed to perform the corresponding operation), resource-level permissions (i.e., whether the user has the permission to process the data to be processed), and other permissions.

[0094] When the verification result obtained through step S1111 is a verification failure, a verification failure handling operation can be executed. The verification failure handling operation can be to return an error response, inform the reason for the verification failure; record the detailed information of the verification failure; in sensitive scenarios (such as when a critical resource is frequently attempted to be accessed), send an alarm message to the administrator or relevant personnel, etc.

[0095] When the verification result obtained through step S1111 is a verification success, execute step S1112.

[0096] Step S1112, record the data operation request.

[0097] Steps S1111 to S1112 provide a solid security guarantee for data processing through a permission verification mechanism, improving the reliability of data processing.

[0098] Furthermore, for the data processing method provided by the present disclosure, after step S140, it may further include step S150 as Figure 8 shown.

[0099] Step S150, generate a data adoption record for the data to be adopted.

[0100] In some embodiments of the present disclosure, the data adoption record in step S150 is a record of all key information during the adoption process of the data to be processed, and may include data source, adoption time, adoption rules, adoption reasons, etc.

[0101] Taking the data related to real estate as an example of the data to be processed, the data adoption record may include: primary key ID, data historical version ID (a unique identifier used to uniquely identify the previous version of this field), entity ID, entity type, entity field, operation ID, adoption factor (used to describe the basis for obtaining the data to be adopted), adoption time (i.e., the time when the data to be adopted is obtained), adoption effective status (used to describe whether the data to be adopted has taken effect), policy configuration used for adoption (used to describe the rules configured when obtaining the adoption policy), adopted policy type (used to describe the category of the rules used to determine the data to be adopted), policy version (used to describe the version of the rules used to determine the data to be adopted), user ID, execution status, etc.

[0102] Generating a data adoption record through step S150 can facilitate tracing the data source, verifying the adoption decision, optimizing the adoption rules, while supporting audit and compliance requirements, and ensuring the transparency and reliability of data processing.

[0103] Regarding step S120, in some embodiments of the present disclosure, it may include steps S127 to S128 as Figure 9 shown.

[0104] Step S127, determining whether the data to be processed needs to be processed based on the adoption policy rules.

[0105] In some embodiments of the present disclosure, a data processing policy may be preset, that is, the processing method of data is defined in advance according to business requirements and data characteristics, including: which data needs to be processed through the adoption policy rules, and which data does not need to be processed through the adoption policy rules and can be directly written.

[0106] When it is determined through step S127 that the data to be processed needs to be processed based on the adoption policy rules, step S128 is executed.

[0107] Step S128, obtaining historical data related to the data to be processed.

[0108] Regarding step S120, when it is determined through step S127 that the data to be processed does not need to be processed based on the adoption policy rules, it may further include step S129 as Figure 10 shown.

[0109] Step S129, writing the data to be processed as the data to be adopted into the target database.

[0110] Through steps S127 to S129, hierarchical management of data processing can be achieved, ensuring the efficient use of system resources, while guaranteeing data quality and the accuracy of business logic.

[0111] Regarding step S130, in some embodiments of the present disclosure, it may include steps S131 to S135 as Figure 11 shown.

[0112] Step S131, obtaining the data source scores of the data to be processed and historical data.

[0113] In some embodiments of the present disclosure, the data source scores of the data to be processed and historical data obtained through step S131 can reflect the data source quality of the data to be processed and historical data. The data source score can be pre-generated.

[0114] Step S132, obtaining target data from the data to be processed and historical data according to the high or low data source scores.

[0115] In some embodiments of the present disclosure, step S132 may first sort the data to be processed and historical data according to the data source scores from high to low; then obtain the target data with the highest data source score from the data to be processed and historical data. When obtaining the target data with the highest data source score, it can be obtained according to a preset number, preset ratio, etc.

[0116] When obtaining target data through step S132, the target data can be directly obtained from all data (i.e., all data to be processed and historical data); in particular, in order to improve the acquisition efficiency of the target data, invalid data can also be filtered from all data first, and then the target data can be obtained from the filtered data. Among them, invalid data can be data that cannot be used continuously due to reasons such as data update, business rule change, error correction, etc.

[0117] Step S133, determining whether the number of target data is equal to 1.

[0118] In some embodiments of the present disclosure, when it is determined through step S133 that the number of target data is not equal to 1, step S134 is executed; when it is determined through step S133 that the number of target data is equal to 1, step S135 is executed.

[0119] Step S134, obtaining the data to be adopted from the target data based on the adoption strategy rule.

[0120] In some embodiments of the present disclosure, when obtaining the data to be adopted from the target data based on the adoption strategy rule through step S134, the adoption strategy rule may include multiple conditions, and the data is screened based on each condition in turn and it is determined whether the screening result can be used as the data to be adopted; if it can, the screening result is directly used as the data to be adopted; if not, it is transferred to the next condition, and the above screening and judgment process is repeated until the data screened based on a certain condition can be used as the data to be adopted.

[0121] Step S135, use the target data as the data to be adopted.

[0122] By obtaining the data to be adopted through steps S131 to S135, it is possible to select higher-quality data from multiple data, improving the credibility and quality of data processing.

[0123] In some embodiments of the present disclosure, in order to make the definition of the adoption strategy rule more flexible and facilitate dynamic modification and extension, the adoption strategy rule can be implemented through a rule engine or a custom expression. For simple and standardized business scenarios, the adoption strategy rule can be a general unified rule; for complex business scenarios, the adoption strategy rule can be configured separately based on different business scenarios to improve the quality of data adoption. During the data processing process, the effectiveness of the adoption strategy rule can also be monitored and evaluated in real time, improving the accuracy and scientific nature of decision-making while ensuring data quality.

[0124] Specifically, the adoption strategy rule can be abstracted into a source node (Source), a calculation node (Calculator), and a write node (Sink); among them, the source node is responsible for providing and obtaining the data source and cleaning rules; the calculation node is responsible for determining the data to be adopted according to the specific adoption strategy rule; the write node is responsible for writing the data to be adopted into the target database. Among them, the calculation node can implement the routing logic during the calculation process through the condition routing router attribute, determining the flow direction of data processing; it flexibly adjusts the data processing path according to conditions. Specifically, the conditional edges of the router attribute can include default and elseTo. default is the default conditional path, and if no other conditions are matched, the data will default to this path; elseTo is used to handle some abnormal situations or data under specific conditions; in the calculation node, the router performs data routing according to multiple conditions, dynamically selects the optimal path, and thus determines the most suitable data to be adopted according to condition judgment and quality evaluation.

[0125] In some embodiments of the present disclosure, when the historical data related to the data to be processed obtained through step S120 is empty, the data to be processed can be directly used as the data to be adopted and then written into the target database through step S140.

[0126] The data processing method provided by the present disclosure realizes data processing through a target database combined with a message middleware, which can ensure data consistency; after testing, the data consistency can reach at least more than 99.999%.

[0127] When the data operation request record fails, an error message can be returned and a retry can be initiated according to a preset retry policy. Since the data processing process has not started yet, it will not cause substantial interference to the data processing flow, thus improving data consistency.

[0128] When the write of the data to be processed into the target database fails, after rollback, an error message can be returned and a retry can be initiated according to a preset retry policy. By rolling back, it is possible to avoid substantial interference with the data processing flow and improve data consistency.

[0129] After writing, when the sending of a message to the message middleware fails, it is possible to wait for the transaction backcheck request of the message middleware, and after receiving the transaction backcheck request, check whether the write to the target database is successful and attempt to resubmit or roll back, thereby improving data consistency through this transaction backcheck request.

[0130] When the retrieval of historical data or data to be adopted fails, a retry can be initiated according to a preset retry policy, and when the retry policy is not met, the relevant information can be placed in the dead letter queue for manual intervention by relevant users such as developers.

[0131] After message submission or rollback, an acknowledgement message can be returned for error handling, monitoring and auditing, taking subsequent actions, etc.; if the return of the acknowledgement message fails, a retry can be performed and idempotency can be ensured so that multiple executions of the same operation will not have additional impacts, thereby improving data consistency.

[0132] Figure 12 An exemplary flowchart implemented based on the data processing method of the present disclosure is shown.

[0133] Figure 12 In the shown flowchart, taking the relevant data of the building dictionary as an example, the data processing method may include: Step S210, obtaining the data to be processed.

[0134] Step S220, controlling the sending of a half-message carrying the data to be processed to the message middleware, and the status of the half-message is pending consumption.

[0135] Step S230, writing the data to be processed into the target database.

[0136] In some embodiments of the present disclosure, when the write of the data to be processed into the target database fails in step S230, step S240 is executed; when the write of the data to be processed into the target database is successful in step S230, step S250 is executed.

[0137] Step S240, in response to the failure of writing the data to be processed, controlling the target database to roll back and controlling the message middleware to roll back the half-message.

[0138] Step S250: In response to successful writing of the data to be processed, control is sent to the message middleware to send a write success message.

[0139] Step S260: Generate a data acquisition record and a data version record for the data to be processed.

[0140] Step S270: In response to the message middleware changing the status of the half-message to consumable according to the write success message, obtain historical data related to the data to be processed.

[0141] Step S280: Obtain the data to be adopted from the data to be processed and the historical data based on the adoption policy rules.

[0142] Step S290: Write the data to be adopted into the target database.

[0143] In some embodiments of the present disclosure, when writing the data to be adopted into the target database through step S290, the data to be adopted can be used to overwrite the data to be processed written through step 230 in the target database.

[0144] Step S291: Generate a data adoption record for the data to be adopted.

[0145] The data processing method provided by the present disclosure preferentially selects high-quality data from the data to be processed and the historical data based on the adoption policy rules and writes it into the target database, which can ensure the stability of the data processing system and improve the quality of the data written into the target database.

[0146] The data processing method provided by the present disclosure can be encapsulated into a Software Development Kit (SDK), so that different business systems or platforms can follow a unified standardized processing process for data processing; by establishing a standardized data processing process and providing a standardized data processing SDK, the data processing success rate can be improved, system failures caused by data errors or non-standard operations can be reduced, and the reliability and scalability can be significantly enhanced. The SDK can be generated based on the metadata information of the target database.

[0147] Through the data acquisition record and the data version record, each submission, modification, and version change of the data can be detailedly recorded. Each data version record can be associated with the corresponding data acquisition record to ensure that specific operation details and time nodes can be traced when needed; this not only improves the transparency of data management but also provides a solid foundation for data quality review and problem tracing.

[0148] Based on any of the above embodiments, the present disclosure further provides a data processing apparatus.

[0149] Figure 13It is a structural schematic block diagram of a data processing device according to an embodiment of the present disclosure.

[0150] As Figure 13 shown, the data processing device includes: A to-be-processed data acquisition module 110, configured to acquire to-be-processed data.

[0151] A historical data acquisition module 120, configured to acquire historical data related to the to-be-processed data.

[0152] A to-be-adopted data acquisition module 130, configured to acquire to-be-adopted data from the to-be-processed data and the historical data based on an adoption policy rule.

[0153] A data writing module 140, configured to write the to-be-adopted data into a target database.

[0154] The above data processing device may be in the form of computer software, and each module of the above data processing device may be implemented by a computer software module.

[0155] For the implementation processes of the functions and roles of each module in the above device, please refer to the implementation processes of the corresponding steps in the above method for details, which will not be elaborated here.

[0156] The execution subject of the data processing method in the specific embodiment of the present disclosure may be a server electronic device.

[0157] Therefore, based on any of the above embodiments, the present disclosure further provides an electronic device, and this electronic device may execute the data processing method of any of the above embodiments described in the present disclosure.

[0158] Figure 14 It is a structural schematic block diagram of an electronic device 1000 according to an embodiment of the present disclosure.

[0159] The hardware structure of the electronic device 1000 may be implemented using a bus architecture. The bus architecture may include any number of interconnected buses and bridges, depending on the specific application of the hardware and the overall design constraints. The bus 1100 connects various circuits including one or more processors 1200, a memory 1300, and / or hardware modules together. The bus 1100 may also connect various other circuits 1400 such as peripheral devices, voltage regulators, power management circuits, external antennas, etc.

[0160] The bus 1100 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of representation, only one connecting line is used in this figure, but it does not mean that there is only one bus or one type of bus.

[0161] The present disclosure also provides a readable storage medium storing a computer program, which when executed by a processor is used to implement the above method. The "readable storage medium" can be any device that can contain, store, communicate, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. More specific examples of the readable storage medium include the following: an electrical connection part with one or more wirings (electronic device), a portable computer diskette (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable read-only memory (CDROM), etc.

[0162] The present disclosure also provides a computer program product. The method of the present disclosure can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed, the processes or functions of the present disclosure are executed in whole or in part.

[0163] The computer program or instructions can be stored in a readable storage medium, or transmitted from one readable storage medium to another. For example, the computer program or instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center in a wired or wireless manner. The readable storage medium can be any available medium that can be accessed, or a data storage device such as a server or data center integrating one or more available mediums. The available medium can be a magnetic medium, such as a floppy disk, a hard disk, or a magnetic tape; it can also be an optical medium, such as a digital video disc; or it can be a semiconductor medium, such as a solid state drive. The computer-readable storage medium can be a volatile or non-volatile storage medium, or can include both volatile and non-volatile types of storage media.

[0164] Those skilled in the art should understand that the embodiments of the present disclosure can be provided as a method, a system, or a computer program product. Therefore, the present disclosure can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present disclosure can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0165] The present disclosure is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to the present disclosure. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices produce means for implementing the functions specified in Figure 1 one or more of the flows Figure 1 or blocks or combinations of blocks.

[0166] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory produce a manufactured article including instruction means that implement the functions specified in Figure 1 one or more of the flows Figure 1 or blocks or combinations of blocks.

[0167] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in Figure 1 one or more of the flows Figure 1 or blocks or combinations of blocks.

[0168] In the description of this specification, the description referring to terms such as "one embodiment / way", "some embodiments / ways", "example", "specific example", or "some examples" means that the specific features, structures, or characteristics described in connection with the embodiment / way or example are included in at least one embodiment / way or example of the present disclosure. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment / way or example. Moreover, the specific features, structures, or characteristics described can be combined in a suitable manner in any one or more embodiments / ways or examples. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments / ways or examples described in this specification and the features of different embodiments / ways or examples.

[0169] In addition, the terms "first" and "second" are used only for descriptive purposes and cannot be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include at least one of such features. In the description of the present disclosure, "a plurality" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0170] Those skilled in the art should understand that the above embodiments are merely for clearly explaining the present disclosure and are not intended to limit the scope of the present disclosure. For those skilled in the art, other changes or variations can be made on the basis of the above disclosure, and these changes or variations are still within the scope of the present disclosure.

Claims

1. A data processing method, characterized in that, Including: Obtain data to be processed; Obtain historical data related to the data to be processed; Obtain data to be adopted from the data to be processed and the historical data based on an adoption policy rule; And Write the data to be adopted into a target database.

2. The data processing method according to claim 1, wherein The obtaining of the historical data related to the data to be processed includes: Control to send a half-message carrying the data to be processed to a message middleware, and the status of the half-message is to be consumed; Write the data to be processed into the target database; In response to successful writing of the data to be processed, control to send a write success message to the message middleware; and In response to the message middleware changing the status of the half-message to consumable according to the write success message, obtain the historical data related to the data to be processed; Optionally, after the response to successful writing of the data to be processed, it further includes: Generate a data acquisition record and a data version record for the data to be processed; Optionally, after writing the data to be processed into the target database, in response to failure of writing the data to be processed, it further includes: Control the target database to roll back and control the message middleware to roll back the half-message.

3. The data processing method according to claim 1 or 2, characterized in that The obtaining of the data to be processed includes: In response to receiving a data operation request, record the data operation request; and Obtain the data to be processed to be operated on according to the data operation request based on the data operation request; Optionally, the recording of the data operation request includes: Perform a permission check on the data operation request to obtain a check result; and When the check result is a successful check, record the data operation request.

4. The data processing method according to claim 1 or 2, characterized in that, After writing the data to be adopted into the target database, it further includes: Generate a data adoption record for the data to be adopted.

5. The data processing method according to claim 1 or 2, characterized in that The obtaining of the historical data related to the data to be processed includes: Judge whether the data to be processed needs to be processed based on the adoption policy rule; and If so, obtain the historical data related to the data to be processed.

6. The data processing method according to claim 5, wherein If not, it further includes: Write the data to be processed into the target database as the data to be adopted.

7. The data processing method according to claim 1 or 2, characterized in that The obtaining of the data to be adopted from the data to be processed and the historical data based on the adoption policy rule includes: Obtain the data source scores of the data to be processed and the historical data; Obtain target data from the data to be processed and the historical data according to the high and low of the data source scores; Judge whether the number of the target data is equal to 1; If not equal to 1, obtain the data to be adopted from the target data based on the adoption policy rule; and If equal to 1, use the target data as the data to be adopted.

8. An electronic device, characterized in that, Including: A memory that stores execution instructions; And A processor that executes the execution instructions stored in the memory, so that the processor executes the data processing method according to any one of claims 1 to 7.

9. A readable storage medium, characterized in that, Execution instructions are stored in the readable storage medium, and when the execution instructions are executed by a processor, they are used to implement the data processing method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the data processing method according to any one of claims 1 to 7.

Images