A data processing method and device, electronic equipment and computer readable medium

Abstract

The application discloses a data processing method and device, electronic equipment and computer readable medium, relates to the automatic programming technical field, and a specific embodiment includes receiving a data processing request, calling a database script sealing pipeline to pull corresponding first program data and second program data, obtaining a database script corresponding to the first program data, and then generating merged program data based on the database script and the second program data, calling an exception detection program to pull a path corresponding to the merged program data, and then determining a data exception item according to the path and outputting. The production problem caused by the fact that program data is not covered and backed up is effectively avoided, the time consumption of program data pulling is shortened, and the efficiency and quality of program data sealing work are improved.

Description

Technical Field

[0001] This application relates to the field of automatic programming technology, and in particular to a data processing method, apparatus, electronic device and computer-readable medium. Background Technology

[0002] Traditional database version packaging mainly involves the following steps: pulling the distributed version control system code, merging and compressing the code, transferring the code, manually checking the code, executing the script file, manually checking the script execution, and repeating the above steps until the final version is packaged. The previous process was time-consuming and inefficient due to the manual pulling, compressing, and transferring of the distributed version control system code; it also suffered from inconsistent script naming and the potential for overlooking crucial file information during manual checks. Summary of the Invention

[0003] In view of this, embodiments of this application provide a data processing method, apparatus, electronic device, and computer-readable medium, which can solve the problem that existing processes have long processing times and low efficiency due to manually pulling distributed version control system code and compressing and transmitting it.

[0004] To achieve the above objectives, according to one aspect of the embodiments of this application, a data processing method is provided, comprising:

[0005] Receive data processing requests and call the database script sealing pipeline to pull the corresponding first program data and second program data;

[0006] Obtain the database script corresponding to the first program data, and then generate merged program data based on the database script and the second program data;

[0007] The anomaly detection program is invoked to retrieve the path corresponding to the merged program data, and then the data anomalies are identified and output based on the path.

[0008] Optionally, the database script sealing pipeline is invoked to retrieve the corresponding first program data and second program data, including:

[0009] Obtain the first cover version date corresponding to the first program data and the second cover version date corresponding to the second program data;

[0010] Retrieve the corresponding first program data and second program data based on the first and second cover version dates.

[0011] Optionally, before obtaining the database script corresponding to the first program data, the method further includes:

[0012] Determine the purpose of the database script;

[0013] Name the database scripts according to their purpose.

[0014] Optionally, the database script corresponding to the first program data is obtained, including:

[0015] Locate and retrieve the database script from the first program data based on the preset naming prefix.

[0016] Optionally, merge procedure data is generated, including:

[0017] Determine the directory in the second program data that corresponds to the database script;

[0018] Place the database script in the directory to generate the merge program data.

[0019] Optionally, determine the directory in the second program data that corresponds to the database script, including:

[0020] Get the name of the database script;

[0021] The names are matched against the directories in the second program data to determine the directories corresponding to the database scripts based on the matching results.

[0022] Optionally, the path corresponding to the merged program data can be retrieved, including:

[0023] Retrieve the backup path, restore path, update path, and compilation / execution path corresponding to the merged program data.

[0024] Optionally, data anomalies can be determined based on the path, including:

[0025] Based on the path, determine the date information of the table creation statement corresponding to the merged program data;

[0026] Obtain the sealing version date corresponding to the merged program data, then match the date information with the sealing version date, and identify the corresponding table creation statement when the match fails as a data anomaly.

[0027] In addition, this application also provides a data processing apparatus, including:

[0028] The receiving unit is configured to receive data processing requests and call the database script sealing pipeline to pull the corresponding first program data and second program data.

[0029] The merged program data generation unit is configured to obtain the database script corresponding to the first program data, and then generate merged program data based on the database script and the second program data.

[0030] The data processing unit is configured to call the anomaly detection program to retrieve the path corresponding to the merged program data, and then determine and output the data anomalies based on the path.

[0031] Optionally, the receiving unit is further configured to:

[0032] Obtain the first cover version date corresponding to the first program data and the second cover version date corresponding to the second program data;

[0033] Retrieve the corresponding first program data and second program data based on the first and second cover version dates.

[0034] Optionally, the device also includes a naming unit, configured to:

[0035] Determine the purpose of the database script;

[0036] Name the database scripts according to their purpose.

[0037] Optionally, the merge program data generation unit is further configured to:

[0038] Locate and retrieve the database script from the first program data based on the preset naming prefix.

[0039] Optionally, the merge program data generation unit is further configured to:

[0040] Determine the directory in the second program data that corresponds to the database script;

[0041] Place the database script in the directory to generate the merge program data.

[0042] Optionally, the merge program data generation unit is further configured to:

[0043] Get the name of the database script;

[0044] The names are matched against the directories in the second program data to determine the directories corresponding to the database scripts based on the matching results.

[0045] Optionally, the data processing unit is further configured to:

[0046] Retrieve the backup path, restore path, update path, and compilation / execution path corresponding to the merged program data.

[0047] Optionally, the data processing unit is further configured to:

[0048] Based on the path, determine the date information of the table creation statement corresponding to the merged program data;

[0049] Obtain the sealing version date corresponding to the merged program data, then match the date information with the sealing version date, and identify the corresponding table creation statement when the match fails as a data anomaly.

[0050] In addition, this application also provides a data processing electronic device, including: one or more processors; and a storage device for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement the data processing method described above.

[0051] In addition, this application also provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the data processing method described above.

[0052] To achieve the above objectives, according to another aspect of the embodiments of this application, a computer program product is provided.

[0053] A computer program product according to an embodiment of this application includes a computer program that, when executed by a processor, implements the data processing method provided in an embodiment of this application.

[0054] One embodiment of the above invention has the following advantages or beneficial effects: This application receives a data processing request, calls a database script sealing pipeline to pull corresponding first program data and second program data; obtains the database script corresponding to the first program data, and then generates merged program data based on the database script and second program data; calls an anomaly detection program to pull the path corresponding to the merged program data, and then determines and outputs data anomalies based on the path. This effectively avoids production problems caused by incomplete program data coverage and backup, shortens the time spent pulling program data, and improves the efficiency and quality of program data sealing work.

[0055] The further effects of the aforementioned unconventional alternative methods will be explained below in conjunction with specific implementation methods. Attached Figure Description

[0056] The accompanying drawings are provided to better understand this application and do not constitute an undue limitation thereof. Wherein:

[0057] Figure 1 This is a schematic diagram of the main flow of a data processing method according to an embodiment of this application;

[0058] Figure 2 This is a schematic diagram of the main flow of a data processing method according to an embodiment of this application;

[0059] Figure 3 This is a schematic flowchart of a data processing method according to an embodiment of this application;

[0060] Figure 4 This is a schematic flowchart of a data processing method according to an embodiment of this application;

[0061] Figure 5This is a schematic diagram of the main units of a data processing apparatus according to an embodiment of this application;

[0062] Figure 6 This is an exemplary system architecture diagram to which embodiments of this application can be applied;

[0063] Figure 7 This is a schematic diagram of the structure of a computer system suitable for implementing terminal devices or servers in the embodiments of this application. Detailed Implementation

[0064] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of this application, including various details to aid understanding. These embodiments should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description. It should be noted that the collection, analysis, use, transmission, and storage of user personal information involved in the technical solutions of this application comply with relevant laws and regulations, are used for legitimate and reasonable purposes, are not shared, disclosed, or sold outside of these legitimate uses, and are subject to supervision and management by regulatory authorities. Necessary measures should be taken to prevent unauthorized access to such personal information data, ensure that personnel authorized to access personal information data comply with relevant laws and regulations, and ensure the security of user personal information. Once this user personal information data is no longer needed, the risk should be minimized by restricting or even prohibiting data collection and / or deleting the data.

[0065] When used, including in certain relevant applications, data is deidentified to protect user privacy, for example by removing specific identifiers, controlling the amount or specificity of stored data, controlling how data is stored, and / or other methods.

[0066] Figure 1 This is a schematic diagram of the main flow of a data processing method according to an embodiment of this application, as shown below. Figure 1 As shown, the data processing methods include:

[0067] Step S101: Receive data processing request and call the database script sealing pipeline to pull the corresponding first program data and second program data.

[0068] In this embodiment, the execution entity of the data processing method (e.g., a server) can receive data processing requests via wired or wireless connections. Specifically, the data processing request can be a request to perform error checking on the program script. This embodiment does not specifically limit the content of the data processing request. Integrated Development Platform (ICDP) Automated Pipeline: The ICDP pipeline executes jobs in the form of containers on a container cloud. The compilation environment involved is instantiated as a container in the form of an image, supporting four programming languages: Java, Node.js, Golang, and Android. This embodiment develops a database script sealing pipeline for the N-PLS system based on ICDP. Specifically, the database script sealing pipeline can be a pipeline composed of various program nodes with temporal characteristics. Each program node can include program nodes for date checking, path checking, format checking, naming checking, etc. This embodiment does not specifically limit the content of the program nodes. The program nodes store the corresponding executable program. The database script is invoked to retrieve the corresponding first program data and second program data for the sealing pipeline. The first program data can be the git code of PLPM_DATABASE, and the second program data can be the git code of PLPM_BATCH.

[0069] Specifically, the database script sealing pipeline is invoked to retrieve the corresponding first program data and second program data, including: obtaining the first sealing version date corresponding to the first program data and the second sealing version date corresponding to the second program data; and retrieving the corresponding first program data and second program data based on the first sealing version date and the second sealing version date.

[0070] For example, configure the Git code for PLPM_BATCH and PLPM_DATABASE, based on the N-PLS system, configure the physical subsystem to N-PLS, configure the code project to PLPM_BATCH_GN, and select the branch to which it belongs based on the release date of each version to achieve automatic retrieval of the corresponding program data.

[0071] Step S102: Obtain the database script corresponding to the first program data, and then generate merged program data based on the database script and the second program data.

[0072] Specifically, before obtaining the database script corresponding to the first program data, the method also includes:

[0073] Determine the purpose of the database script. Specifically, the purpose may include creating new tables or table structures, modifying production data, or using Chinese characters in the table creation script. This application embodiment does not specifically limit the purpose. Name the database script according to its purpose.

[0074] For example, to ensure system standardization and consistency, when naming database scripts, if the script is used to create a new table or table structure (i.e., involves DDL operations), the script should be named `plpm_table`, followed by the table creation operation (create or modify), the table name, the developer who created the table, and the version date, with all of the above separated by underscores "_". If the database script is used to modify production data (involving DML operations), and production table modifications are required, the script should be named `plpm_data`, followed by the table name, the developer who created the table, and the version date, with all of the above separated by underscores "_". When Chinese characters are present in the table creation script (including in the following situations: when describing table fields during table creation, or when inserting data from an existing table), the table encoding format should be UNIX, GB2313 format, to avoid garbled characters.

[0075] Specifically, the database script for obtaining the data from the first program includes:

[0076] Locate and retrieve the database script from the first program data based on the preset naming prefix.

[0077] Specifically, the preset naming prefix can be, for example, SJKJB, used to indicate the start position of the database script. The position of the database script is located by locating the preset naming prefix, thereby obtaining the database script at that position.

[0078] Step S103: Call the anomaly detection program to retrieve the path corresponding to the merged program data, and then determine and output the data anomalies based on the path.

[0079] Specifically, the path corresponding to the data retrieved from the merge process includes:

[0080] Retrieve the backup path, restore path, update path, and compilation / execution path corresponding to the merged program data.

[0081] The executing entity can retrieve the paths to four key files from the merged program data. For example, the four key files are: PLPJK_DB_BAK.sh (used for version package backup, from which the backup path can be extracted), PLPJK_DB_RECOVERY.sh (used for version package restoration, from which the restoration path can be extracted), PLPJK_DB_UPDATE.sh (used for version package updates, from which the update path can be extracted), and UpdProc.sh (used for compiling and executing database table creation statements, determining if it corresponds to the version date, from which the compilation and execution path can be extracted).

[0082] Specifically, data anomalies are identified based on the path, including:

[0083] Based on the path, determine the date information of the table creation statement corresponding to the merged program data; obtain the sealing version date corresponding to the merged program data, and then match the date information with the sealing version date. If the match fails, the corresponding table creation statement is identified as a data anomaly.

[0084] The four key files are: PLPJK_DB_BAK.sh, used for version package backup (from which the backup path can be extracted); PLPJK_DB_RECOVERY.sh, used for version package recovery (from which the recovery path can be retrieved); PLPJK_DB_UPDATE.sh, used for version package updates (from which the update path can be retrieved); and UpdProc.sh, used for compiling and executing database table creation statements, checking if it corresponds to the version date, and retrieving the compilation and execution path. These four key files determine whether an existing production version package can be overwritten. The existence of these four key files (determined by obtaining the corresponding paths) and whether the dates of the table creation statements in these four key files (obtained from the date information of the table creation statements corresponding to the merged program data) correspond to the version date are crucial steps that must be checked. The passing criteria are: all four files must exist, and the dates in all four files must correspond to the version date. If the passing criteria are not met, the data items that fail to meet the criteria (i.e., the table creation statements that fail to match) are identified as abnormal data items.

[0085] This embodiment receives a data processing request, invokes a database script sealing pipeline to retrieve the corresponding first and second program data; obtains the database script corresponding to the first program data, and then generates merged program data based on the database script and the second program data; invokes an anomaly detection program to retrieve the path corresponding to the merged program data, and then determines and outputs data anomalies based on the path. This effectively avoids production problems caused by incomplete program data coverage and backup, shortens the program data retrieval time, and improves the efficiency and quality of the program data sealing process.

[0086] Figure 2 This is a schematic diagram of the main flow of a data processing method according to an embodiment of this application, as shown below. Figure 2 As shown, the data processing methods include:

[0087] Step S201: Receive data processing request and call the database script sealing pipeline to pull the corresponding first program data and second program data.

[0088] Step S202: Obtain the database script corresponding to the first program data.

[0089] Step S203: Determine the directory in the second program data that corresponds to the database script.

[0090] The executing entity can determine the directory in the second program data that corresponds to the database script through a pre-defined database of correspondences between scripts and directories.

[0091] As another implementation, the executing entity can also calculate the similarity between the database script and each directory in the second program data, and determine the directory corresponding to the database script based on the similarity. Specifically, this can be done by converting the database script into a first vector based on word embedding, converting each directory in the second program data into a second vector, and calculating the similarity between the first vector and each second vector. The directory corresponding to the second vector with the highest similarity is then determined as the directory corresponding to the database script.

[0092] Step S204: Place the database script in the directory to generate merge program data.

[0093] Specifically, the executing entity can further determine the subdirectories in the directory corresponding to the database script. If a subdirectory corresponding to the database script exists, the database script is placed in the corresponding subdirectory. If no subdirectory corresponding to the database script exists, the database script is placed in the corresponding directory to generate merged program data.

[0094] Step S205: Call the anomaly detection program to retrieve the path corresponding to the merged program data, and then determine and output the data anomalies based on the path.

[0095] The anomaly detection program can retrieve all paths in the merged program data, or it can retrieve paths with specific prefixes. This embodiment does not specifically limit the paths retrieved by the anomaly detection program from the merged program data. The executing entity can compare each data item in the retrieved path (e.g., the date corresponding to the database table creation statement) with preset data items (e.g., the database version date) to identify data items with discrepancies. These discrepancies are then identified as data anomalies and output. This allows for quick and accurate identification of data anomalies in the data to be processed, improving the efficiency and accuracy of anomaly data investigation.

[0096] Figure 3 This is a schematic diagram of the main flow of a data processing method according to an embodiment of this application, as shown below. Figure 3 As shown, the data processing methods include:

[0097] Step S301: Receive data processing request and call the database script sealing pipeline to pull the corresponding first program data and second program data.

[0098] Data processing requests can be requests to perform error checking on program scripts. This application embodiment does not specifically limit the content of the data processing request. This application embodiment develops a database script sealing pipeline for the N-PLS system based on ICDP. Specifically, the database script sealing pipeline can be a pipeline composed of various program nodes with temporal characteristics. Each program node can include program nodes for date checking, path checking, format checking, naming checking, etc. This application embodiment does not specifically limit the content of the program nodes. The program nodes store the corresponding executable program. The database script sealing pipeline is invoked to retrieve the corresponding first program data and second program data, where the first program data can be the Git code of PLPM_BATCH, and the second program data can be the Git code of PLPM_DATABASE.

[0099] Step S302: Obtain the database script corresponding to the first program data.

[0100] When multiple database scripts exist in the first program data, all database scripts in the first program data can be retrieved. Alternatively, the types of these multiple database scripts can be determined, and the database script to be retrieved can be determined based on the types of the multiple database scripts. Specifically, the types of the multiple database scripts can be matched with the data types of the second program data to retrieve one or more database scripts that match the data types of the second program data.

[0101] Step S303: Obtain the name of the database script.

[0102] After standardizing the database script naming, the database script name can be associated with its purpose. By retrieving the database script name, it can be matched with various directories in the second program data. Each directory in the second program data can then be associated with its purpose.

[0103] Step S304: Match the name with each directory in the second program data to determine the directory corresponding to the database script based on the matching results.

[0104] Specifically, the executing entity can determine the first purpose corresponding to the name of the acquired database script, and determine the second purpose corresponding to each directory in the second program data; by matching the first purpose with each second purpose, the directory corresponding to the matched second purpose is determined as the directory corresponding to the database script.

[0105] Step S305: Place the database script in the directory to generate merge program data.

[0106] Specifically, first, the acquired second program data is compressed, for example, the git code of PLPM_BATCH is compressed, so that the PLPM_BATCH code is automatically acquired and transmitted to the target server. The first program data, such as the database script in the middle of the PLPM_DATABASE code, is put into the corresponding directory of the second program data, such as the PLPM_BATCH code. The code is merged to generate merged program data, and then it is decompressed.

[0107] Step S306: Call the anomaly detection program to retrieve the path corresponding to the merged program data, and then determine and output the data anomalies based on the path.

[0108] Executing an anomaly detection program, such as the script `check_version.sh`, can check for problems in the merged program data. The `check_version.sh` script extracts four key files from the middle of the version package: `PLPJK_DB_BAK.sh` for version package backup, `PLPJK_DB_RECOVERY.sh` for version package recovery, `PLPJK_DB_UPDATE.sh` for version package updates, and `UpdProc.sh` for compiling and executing database table creation statements, checking if they correspond to the version date. For example, after `check_version.sh` finishes running the database script to check the code, it can display the middle code date of the four key script files as the version date, effectively saving time for manual verification, greatly improving the efficiency and accuracy of database sealing, and clearly identifying problems in the merged program data without requiring manual inspection. This largely solves the time-consuming and missed detection problems associated with database script naming conventions, significantly improving sealing efficiency and promoting the standardization of sealing work.

[0109] Figure 4 This is a schematic diagram of the main flow of a data processing method according to an embodiment of this application. Figure 4As shown, the database script sealing pipeline is invoked to pull the PLPM_BATCH and PLPM_database codes. The merged code is generated by adding PLPM table creation statements. Specifically, the database script in the PLPM_database code is added to the corresponding directory of the PLPM_BATCH code using PLPM table creation statements to obtain the merged code, which is the merged program data in this embodiment. The merged code is then compressed, decompressed, and transmitted to the server to become the server's code. An anomaly detection program, such as GnWorkerCheck.java and check_version.sh, is executed on the server to perform script checks and critical file checks, respectively. The GnWorkerCheck.java script only needs to set the PLPM_database code input parameter to the code pull path; setting this as a fixed input parameter will execute the corresponding script checks, which may include checking script naming, encoding format, and time. The `check_version.sh` script extracts four key files from the version package obtained from the merged code (i.e., merged program data): `PLPJK_DB_BAK.sh` for version package backup, `PLPJK_DB_RECOVERY.sh` for version package restoration, `PLPJK_DB_UPDATE.sh` for version package update, and `UpdProc.sh` for compiling and executing database table creation statements, checking if they correspond to the version date. If the script and key file checks pass, the sealing program `Mkplpmver_new.sh` is called to generate the version package. If the script and key file checks fail, the script naming conventions are checked, the corresponding code is modified, and the script is resubmitted to the database script sealing pipeline for further checks until the version package is generated.

[0110] This application embodiment utilizes a database script-based package pipeline to streamline the code retrieval, merging, compression, and transmission process, significantly improving the efficiency of code retrieval and merging. This application embodiment proposes naming conventions for DDL operations, DML operations, and database script formats, avoiding execution problems caused by naming issues in database table creation statements, making the entire process more standardized and rational, and improving communication efficiency. This application embodiment uses the self-developed automated check script GnWorkerCheck.java to check for naming issues in database table creation statements, clearly identifying problems in the database scripts without requiring manual inspection, largely solving the time-consuming and omission problems associated with database script naming conventions, greatly improving package sealing efficiency and promoting the standardization of package sealing work. By calling the self-developed check_version.sh script to check for issues such as dates, paths, and formats in critical files, production problems caused by incomplete code overwriting and backup are effectively avoided, significantly improving the quality of the package sealing work. The ICDP automated pipeline implements the code retrieval, merging, compression, and transmission process, shortening execution time by nearly 85%, greatly improving the efficiency of code retrieval and merging. By calling the self-written automated check scripts GnWorkerCheck.java and check_version.sh, automated checks are performed, effectively avoiding production problems caused by incomplete code coverage and backup, and significantly improving the quality of the board sealing process.

[0111] Figure 5 This is a schematic diagram of the main units of a data processing apparatus according to an embodiment of this application. Figure 5 As shown, the data processing device 500 includes a receiving unit 501, a merging program data generation unit 502, and a data processing unit 503.

[0112] The receiving unit 501 is configured to receive data processing requests and call the database script sealing pipeline to pull the corresponding first program data and second program data.

[0113] The merged program data generation unit 502 is configured to obtain the database script corresponding to the first program data, and then generate merged program data based on the database script and the second program data.

[0114] The data processing unit 503 is configured to call the anomaly detection program to retrieve the path corresponding to the merged program data, and then determine and output the data anomaly items based on the path.

[0115] In some embodiments, the receiving unit 501 is further configured to: obtain the first cover version date corresponding to the first program data and the second cover version date corresponding to the second program data; and retrieve the corresponding first program data and second program data according to the first cover version date and the second cover version date.

[0116] In some embodiments, the data processing apparatus further includes Figure 5 The naming unit, not shown, is configured to: determine the purpose of the database script; and name the database script according to its purpose.

[0117] In some embodiments, the merged program data generation unit 502 is further configured to: locate and obtain the database script in the first program data according to a preset naming prefix.

[0118] In some embodiments, the merged program data generation unit 502 is further configured to: determine the directory corresponding to the database script in the second program data; and place the database script in the directory to generate merged program data.

[0119] In some embodiments, the merged program data generation unit 502 is further configured to: obtain the name of the database script; match the name with each directory in the second program data to determine the directory corresponding to the database script based on the matching result.

[0120] In some embodiments, the data processing unit 503 is further configured to: retrieve the backup path, recovery path, update path, and compilation / execution path corresponding to the merge program data.

[0121] In some embodiments, the data processing unit 503 is further configured to: determine the date information of the table creation statement corresponding to the merged program data according to the path; obtain the sealing version date corresponding to the merged program data, and then match the date information with the sealing version date, and determine the table creation statement corresponding to the failure of the match as a data anomaly.

[0122] It should be noted that the data processing method and data processing device in this application are related in specific implementation, so repeated content will not be described again.

[0123] Figure 6 An exemplary system architecture 600 is shown that can be applied to the data processing method or data processing apparatus of the embodiments of this application.

[0124] like Figure 6As shown, system architecture 600 may include terminal devices 601, 602, and 603, a network 604, and a server 605. Network 604 serves as the medium for providing communication links between terminal devices 601, 602, and 603 and server 605. Network 604 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0125] Users can use terminal devices 601, 602, and 603 to interact with server 605 via network 604 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 601, 602, and 603, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0126] Terminal devices 601, 602, and 603 can be various electronic devices with data processing screens and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0127] Server 605 can be a server providing various services, such as a backend management server (for example only) supporting data processing requests submitted by users using terminal devices 601, 602, and 603. The backend management server can receive data processing requests, invoke the database script sealing pipeline to retrieve the corresponding first and second program data; obtain the database script corresponding to the first program data, and then generate merged program data based on the database script and the second program data; invoke the anomaly detection program to retrieve the path corresponding to the merged program data, and then determine and output data anomalies based on the path. This effectively avoids production problems caused by incomplete program data coverage and backup, shortens the time spent retrieving program data, and improves the efficiency and quality of program data sealing.

[0128] It should be noted that the data processing method provided in this application embodiment is generally executed by server 605, and correspondingly, the data processing device is generally located in server 605.

[0129] It should be understood that Figure 6 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0130] The following is for reference. Figure 7 It shows a schematic diagram of the structure of a computer system 700 suitable for implementing a terminal device according to the embodiments of this application. Figure 7 The terminal device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0131] like Figure 7 As shown, the computer system 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 702 or programs loaded from storage section 708 into random access memory (RAM) 703. The RAM 703 also stores various programs and data required for the operation of the computer system 700. The CPU 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0132] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 710 as needed so that computer programs read from it can be installed into the storage section 708 as needed.

[0133] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 709, and / or installed from removable medium 711. When the computer program is executed by central processing unit (CPU) 701, it performs the functions defined above in the system of this application.

[0134] It should be noted that the computer-readable medium shown in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. Computer-readable storage media can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0135] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0136] The units described in the embodiments of this application can be implemented in software or hardware. The described units can also be located in a processor; for example, a processor can be described as including a receiving unit, a merging program data generation unit, and a data processing unit. The names of these units do not necessarily limit the specific unit itself.

[0137] In another aspect, this application also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs that, when executed by the device, cause the device to receive a data processing request, invoke a database script assembly pipeline to retrieve corresponding first program data and second program data; obtain the database script corresponding to the first program data, and then generate merged program data based on the database script and the second program data; invoke an anomaly detection program to retrieve the path corresponding to the merged program data, and then determine and output data anomalies based on the path.

[0138] The computer program product of this application includes a computer program that, when executed by a processor, implements the data processing method in the embodiments of this application.

[0139] According to the technical solution of the embodiments of this application, production problems caused by incomplete program data coverage and backup are effectively avoided, the time spent pulling program data is shortened, and the efficiency and quality of program data sealing work are improved.

[0140] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A data processing method, characterized in that, include: Receive data processing requests and call the database script sealing pipeline to pull the corresponding first program data and second program data; Obtain the database script corresponding to the first program data, and then generate merged program data based on the database script and the second program data; The process of generating merged program data includes: obtaining the name of the database script; matching the name with each directory in the second program data to determine the directory corresponding to the database script based on the matching results; and placing the database script in the directory to generate merged program data. The anomaly detection program is invoked to retrieve the path corresponding to the merged program data, and then the data anomalies are determined and output based on the path.

2. The method according to claim 1, characterized in that, The process of calling the database script sealing pipeline to retrieve the corresponding first program data and second program data includes: Obtain the first cover version date corresponding to the first program data and the second cover version date corresponding to the second program data; Retrieve the corresponding first program data and second program data based on the first and second cover version dates.

3. The method according to claim 1, characterized in that, Before obtaining the database script corresponding to the first program data, the method further includes: Determine the purpose of the database script; Name the database scripts according to their intended use.

4. The method according to claim 1, characterized in that, The database script for obtaining the data corresponding to the first program includes: The database script in the first program data is located and obtained based on the preset naming prefix.

5. The method according to claim 1, characterized in that, The path for retrieving the merged program data includes: Retrieve the backup path, recovery path, update path, and compilation / execution path corresponding to the merged program data.

6. The method according to claim 1, characterized in that, The step of determining data anomalies based on the path includes: Based on the path, determine the date information of the table creation statement corresponding to the merged program data; Obtain the sealing version date corresponding to the merged program data, and then match the date information with the sealing version date. If the match fails, the corresponding table creation statement is identified as a data anomaly.

7. A data processing apparatus, characterized in that, include: The receiving unit is configured to receive data processing requests and call the database script sealing pipeline to pull the corresponding first program data and second program data. The merged program data generation unit is configured to obtain the database script corresponding to the first program data, and then generate merged program data based on the database script and the second program data; the merged program data generation unit is further configured to: obtain the name of the database script; The name is matched with each directory in the second program data to determine the directory corresponding to the database script based on the matching results; Place the database script in the directory to generate merged program data; The data processing unit is configured to call an anomaly detection program to retrieve the path corresponding to the merged program data, and then determine and output the data anomalies based on the path.

8. The apparatus according to claim 7, characterized in that, The receiving unit is further configured to: Obtain the first cover version date corresponding to the first program data and the second cover version date corresponding to the second program data; Retrieve the corresponding first program data and second program data based on the first and second cover version dates.

9. The apparatus according to claim 7, characterized in that, The device also includes a naming unit, configured to: Determine the purpose of the database script; Name the database scripts according to their intended use.

10. The apparatus according to claim 7, characterized in that, The merging process data generation unit is further configured to: The database script in the first program data is located and obtained based on the preset naming prefix.

11. A data processing electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-6.

12. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-6.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-6.

Images