Data processing method and device, computer device and storage medium

Abstract

The application relates to the technical field of big data, and provides a data processing method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: in the case that a preset task comparison condition is triggered, extracting an online development task data set and a production scheduling development task data set from different data sources respectively; intercepting online task identifiers in the online development task data set and production scheduling task identifiers in the production scheduling development task data set to obtain an online task identifier set and a production scheduling task identifier set; and performing consistency comparison on the online task identifier set and the production scheduling task identifier set to obtain a task comparison result. The method can efficiently and accurately compare whether the online task and the production scheduling task are consistent, quickly detect task identifiers that are missing or repeated in the online task and the production scheduling task, and improve the development task comparison efficiency.

Description

Technical Field

[0001] This application relates to the field of big data technology, and in particular to a data processing method, apparatus, computer equipment, storage medium, and computer program product. Background Technology

[0002] In large technology companies, due to real-time changes in regulatory policies and customer needs, development tasks in the production scheduling system may be canceled or delayed at any time. Therefore, the production system needs to make real-time adjustments based on the production scheduling system. Failure to adjust in a timely manner will lead to a discrepancy between the actual deployment scope and the scheduled scope, resulting in too many or too few development tasks being deployed, thus causing production problems. Therefore, ensuring consistency between the deployment scope and the scheduled scope is particularly important.

[0003] Currently, the production scheduling system is mainly used by requirements analysts and developers, while the production deployment system is mainly used by production schedulers and operations and testing personnel. The two systems are independent of each other; therefore, during actual production deployment, a designated person manually compares the online and production deployment scopes exported from the two systems.

[0004] However, due to the large number of development tasks each time and the frequent changes in deployment requirements, manual comparison suffers from low accuracy and low efficiency. Summary of the Invention

[0005] Therefore, it is necessary to provide a data processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can improve comparison efficiency in response to the above-mentioned technical problems.

[0006] Firstly, this application provides a data processing method. The method includes:

[0007] When the preset task comparison conditions are triggered, the online development task dataset and the production scheduling development task dataset are extracted from different data sources respectively.

[0008] Extract the launch task identifier from the launch development task dataset and the production scheduling task identifier from the production scheduling development task dataset to obtain the launch task identifier set and the production scheduling task identifier set;

[0009] The consistency of the online task identifier set and the production scheduling task identifier set is compared to obtain the task comparison results.

[0010] In one embodiment, the online task identifiers from the online development task dataset and the production task identifiers from the production scheduling task dataset are extracted to obtain the online task identifier set and the production task identifier set, including:

[0011] Extract the launch task identifier from the launch development task dataset and the production scheduling task identifier from the production scheduling development task dataset;

[0012] The online task identifier and production scheduling task identifier data are stored in different data tables, resulting in an online task identifier column and a production scheduling task identifier data column. The online task identifier set includes the online task identifier column, and the production scheduling task identifier set includes the production scheduling task identifier column.

[0013] In one embodiment, the online task identifier includes the first physical subsystem number and the online task number, and the production scheduling task identifier data includes the second physical subsystem number and the production scheduling task number.

[0014] The online task identifier and production scheduling task identifier data are stored separately in different data tables, resulting in the following columns:

[0015] The first physical subsystem number and the online task number are concatenated, and the concatenated first physical subsystem number and online task number are stored in the data column of the pre-built data table to obtain the online task identifier column;

[0016] The second physical subsystem number and the production scheduling task number are concatenated and stored in a data column of a pre-built data table to obtain the production scheduling task identifier data column.

[0017] In one embodiment, the consistency comparison between the online task identifier set and the production scheduling task identifier set includes:

[0018] If an abnormal online task identifier is detected in the online task identifier set, the abnormal online task identifier is marked using the first identifier method. The abnormal online task identifier is development task identifier data that does not exist in the production scheduling task identifier set.

[0019] If an abnormal production scheduling task identifier is detected in the production scheduling task identifier set, the abnormal production scheduling task identifier is marked using the second marking method. The abnormal production scheduling task identifier is the development task identifier data that does not exist in the online task identifier set.

[0020] In one embodiment, before performing a consistency comparison between the online task identifier set and the production scheduling task identifier set, the method further includes:

[0021] Data preprocessing is performed on the online task identifier set and the production scheduling task identifier set. Data preprocessing includes at least one of data deduplication, data format standardization, and naming standardization.

[0022] The consistency comparison between the online task identifier set and the production scheduling task identifier set includes:

[0023] Perform a consistency comparison between the online task identifier set and the production scheduling task identifier set after data preprocessing.

[0024] In one embodiment, a consistency comparison is performed between the preprocessed online task identifier set and the production scheduling task identifier set to obtain the task comparison results, including:

[0025] Sort the preprocessed set of online task identifiers and the set of production task identifiers according to the online task identifier and the production task identifier;

[0026] The consistency comparison between the sorted online task identifier set and the production scheduling task identifier set is performed to obtain the task comparison results.

[0027] In one embodiment, the preset task comparison conditions include any one of the following:

[0028] The first requirement is to reach the preset development task comparison cycle;

[0029] The second item is receiving a click on the task comparison button, which is built based on the Visual Basic editor of Excel.

[0030] In one embodiment, after obtaining the task comparison results, the method further includes:

[0031] Use preset visualization tools to visualize the task comparison results;

[0032] The task comparison results include task priority, task dependencies, and abnormal task identification results.

[0033] Secondly, this application also provides a data processing apparatus. The apparatus includes:

[0034] The data extraction module is used to extract the online development task dataset and the production scheduling development task dataset from different data sources when the preset task comparison conditions are triggered.

[0035] The data extraction and processing module is used to extract the online task identifier from the online development task dataset and the production scheduling task identifier from the production scheduling development task dataset to obtain the online task identifier set and the production scheduling task identifier set.

[0036] The data comparison module is used to perform consistency comparison between the online task identifier set and the production scheduling task identifier set to obtain the task comparison results.

[0037] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0038] When the preset task comparison conditions are triggered, the online development task dataset and the production scheduling development task dataset are extracted from different data sources respectively.

[0039] Extract the launch task identifier from the launch development task dataset and the production scheduling task identifier from the production scheduling development task dataset to obtain the launch task identifier set and the production scheduling task identifier set;

[0040] The consistency of the online task identifier set and the production scheduling task identifier set is compared to obtain the task comparison results.

[0041] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0042] When the preset task comparison conditions are triggered, the online development task dataset and the production scheduling development task dataset are extracted from different data sources respectively.

[0043] Extract the launch task identifier from the launch development task dataset and the production scheduling task identifier from the production scheduling development task dataset to obtain the launch task identifier set and the production scheduling task identifier set;

[0044] The consistency of the online task identifier set and the production scheduling task identifier set is compared to obtain the task comparison results.

[0045] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0046] When the preset task comparison conditions are triggered, the online development task dataset and the production scheduling development task dataset are extracted from different data sources respectively.

[0047] Extract the launch task identifier from the launch development task dataset and the production scheduling task identifier from the production scheduling development task dataset to obtain the launch task identifier set and the production scheduling task identifier set;

[0048] The consistency of the online task identifier set and the production scheduling task identifier set is compared to obtain the task comparison results.

[0049] The aforementioned data processing method, apparatus, computer equipment, storage medium, and computer program product, when triggered by preset task comparison conditions, extract online development task datasets and production scheduling development task datasets from different data sources. Then, they extract the online task identifiers from the online development task dataset and the production scheduling task identifiers from the production scheduling task dataset to obtain online task identifier sets and production scheduling task identifier sets. Finally, they perform a consistency comparison between the online task identifier sets and the production scheduling task identifier sets to obtain the task comparison results. This entire solution connects the previously isolated production system and the production scheduling system. By pre-setting task comparison conditions, it enables automatic comparison of development tasks. By extracting online task identifiers and production scheduling task identifiers and comparing the online task identifier sets and production scheduling task identifier sets, it can efficiently and accurately compare whether online tasks and production tasks are consistent, quickly detecting any missing or duplicate task identifiers in the online tasks and production tasks, thus improving the efficiency of development task comparison. Furthermore, adopting the above solution also makes it easier for operators to promptly identify abnormal tasks and add or delete tasks in the online scope, which helps to ensure the consistency between the online task scope and the production scheduling task scope, and improves the data security of the production environment. Attached Figure Description

[0050] Figure 1 This is a diagram illustrating the application environment of a data processing method in one embodiment.

[0051] Figure 2 This is a flowchart illustrating a data processing method in one embodiment;

[0052] Figure 3 This is a flowchart illustrating a data processing method in one embodiment;

[0053] Figure 4 This is a flowchart illustrating the data processing method in another embodiment;

[0054] Figure 5 This is a flowchart illustrating the data processing method in yet another embodiment;

[0055] Figure 6 This is a detailed flowchart illustrating a data processing method in one embodiment;

[0056] Figure 7 This is a detailed flowchart illustrating the data processing method in another embodiment;

[0057] Figure 8 This is a structural block diagram of a data processing device in one embodiment;

[0058] Figure 9 This is a structural block diagram of the data processing apparatus in another embodiment;

[0059] Figure 10This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0061] It should be noted that the data involved in this application (including but not limited to data used for analysis, stored online task tag sets and production scheduling task identifier sets, and displayed task comparison results data) are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0062] The data processing method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or placed on a cloud or other network server. Specifically, an operator can operate the terminal 102's interface to deploy a development task (hereinafter referred to as a task) comparison program on server 104 and start the program. The program has pre-set task comparison conditions. Server 104 continuously monitors whether the current program meets the preset task comparison conditions. If the preset task comparison conditions are met, the development task comparison logic is executed, extracting the online development task dataset and the production scheduling development task dataset from different data sources. Then, the online task identifier in the online development task dataset and the production scheduling task identifier in the production scheduling development task dataset are extracted to obtain the online task identifier set and the production scheduling task identifier set. Finally, a consistency comparison is performed on the online task identifier set and the production scheduling task identifier set to obtain the task comparison result.

[0063] The terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, and smart in-vehicle systems. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted devices. The server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0064] In one embodiment, such as Figure 2 As shown, a data processing method is provided, which can be applied to... Figure 1 Taking the server in the example, the following steps are included:

[0065] S200 determines in real time whether preset task comparison conditions can be triggered.

[0066] In this embodiment, the preset task comparison conditions may include, but are not limited to, timed task triggering conditions. The data source may be the production system and the scheduling system. In specific implementation, operators may pre-configure the task comparison conditions and the corresponding data source paths, edit the task comparison program, deploy the task comparison program on the server, start the task comparison program, and the server will determine in real time whether the current situation meets the preset task comparison conditions.

[0067] S400, when the preset task comparison conditions are triggered, extracts the online development task dataset and the production scheduling development task dataset from different data sources respectively.

[0068] The deployment development task dataset includes basic information on multiple development tasks ready for production deployment. Specifically, the basic information for each development task includes the physical subsystem number (a unique identifier for the physical subsystem to which the development task belongs), the development task number, the development task description, the developers, the priority, the task status, the numbers or descriptions of the requirements or functionalities associated with the development task, and links or paths to documents or materials related to the development task.

[0069] The production scheduling development task dataset includes basic information on multiple development tasks that are rationally planned and scheduled based on product release plans and resource availability. Specifically, the basic information of a development task includes the planned or actual launch date, physical subsystem number (a unique number identifying the physical subsystem to which the development task belongs), development task number, development task description, developers, priority, task status, the number or description of the requirements or functionalities associated with the development task, and links or paths to documents or materials related to the development task.

[0070] The data source can include the production system and the production scheduling system, or it can be a file or database that stores the scope of the production scheduling task and the scope of the production task.

[0071] In specific implementation, taking the production system and scheduling system as data sources as an example, the server can continuously monitor whether the current situation meets the task comparison conditions. When the preset task comparison conditions are triggered, the production system access path, the file path of the online task scope, and the name of the online task worksheet, the scheduling system access path, the file path of the scheduling task scope, and the name of the scheduling task worksheet, etc., can be used to extract the online development task dataset and the scheduling development task dataset from the production system and the scheduling system, respectively.

[0072] Alternatively, in other embodiments, the online development task dataset and the production scheduling development task dataset can be extracted from the production system and the production scheduling system using big data processing tools.

[0073] S600 extracts the online task identifier from the online development task dataset and the production scheduling task identifier from the production scheduling development task dataset to obtain the online task identifier set and the production scheduling task identifier set.

[0074] The identifier for a development task to be launched includes, but is not limited to, the physical subsystem number, development task number, and task name of the development task to be launched. The identifier for a production task includes, but is not limited to, the physical subsystem number, development task number, and task name of the production task.

[0075] Following the previous embodiment, after obtaining the online development task dataset and the production scheduling development task dataset, the online task identifiers and production scheduling task identifiers can be extracted from the online development task dataset and the production scheduling development task dataset, respectively, to obtain the online task identifier set and the production scheduling task identifier set. In practical applications, the field containing the development task identifier usually has a specific column name or identifier in the dataset, and the online development task dataset and the production scheduling development task dataset exist in the form of text files or database tables. Therefore, in this embodiment, the values ​​of the task identifier fields in the above datasets can be extracted one by one using a programming language or tool, such as Python's pandas library or Excel formulas, to obtain the online task identifiers and production scheduling task identifiers. Then, the extracted online task identifiers and production scheduling task identifiers are collected separately to obtain the online task identifier set and the production scheduling task identifier set. In another embodiment, the online task identifiers and production scheduling task identifiers can also be extracted by using a delimiter. For example, if the task identifier is separated from other fields by an underscore ("_"), and the task identifier is located before the underscore ("Task Identifier_XXX"), then you can locate the underscore and extract the field before the underscore to obtain the task identifier.

[0076] S800 performs a consistency comparison between the online task identifier set and the production scheduling task identifier set to obtain the task comparison results.

[0077] Following the previous embodiment, after obtaining the online task identifier set and the production scheduling task identifier set, a consistency comparison can be performed between them. Specifically, matching can be performed using unique identifiers, i.e., task identifiers. That is, for each task identifier in the online task identifier set, it is matched against the production scheduling task identifier set to find matching relationships; similarly, for each task identifier in the production scheduling task identifier set, it is matched against the online task identifier set to find matching relationships, thereby detecting development tasks with discrepancies. During the comparison process, the matching results are recorded. Specifically, a comparison result table or file can be created to record the matching relationships between the online task identifier set and the production scheduling task identifier set line by line. The comparison results can include the status of the online task identifiers, the status of the production scheduling task identifiers, and the corresponding relationships between them. During the comparison process, some inconsistencies may occur, such as no matching items being found or multiple possible matching items existing. Appropriate measures can be taken for these situations. Manual confirmation, further investigation, or consultation with relevant personnel can be conducted to determine the final comparison results, providing accurate data support for subsequent task management and coordination.

[0078] In the above data processing method, when the preset task comparison conditions are triggered, the online development task dataset and the production scheduling development task dataset are extracted from different data sources. Then, the online task identifiers from the online development task dataset and the production scheduling task identifiers from the production scheduling development task dataset are extracted to obtain the online task identifier set and the production scheduling task identifier set. Finally, the consistency comparison of the online task identifier set and the production scheduling task identifier set is performed to obtain the task comparison result. The entire solution connects the originally isolated production system and the production scheduling system. By pre-setting task comparison conditions, it can realize the automatic comparison of development tasks. By extracting the online task identifiers and production scheduling task identifiers and comparing the online task identifier set and the production scheduling task identifier set, it can efficiently and accurately compare whether the online tasks and production tasks are consistent, quickly detect missing or duplicate task identifiers in the online tasks and production tasks, and improve the efficiency of development task comparison. Furthermore, adopting the above solution also makes it easier for operators to promptly identify abnormal tasks and add or delete tasks in the online scope, which helps to ensure the consistency between the online task scope and the production scheduling task scope, and improves the data security of the production environment.

[0079] In one embodiment, the preset task comparison conditions include any one of the following:

[0080] The first requirement is to reach the preset development task comparison cycle;

[0081] The second item is receiving a click on the task comparison button, which is built based on the Visual Basic editor of Excel.

[0082] In practical applications, due to the large number of development tasks deployed each time and the frequent changes in deployment requirements, a development task comparison scheme that can meet real-time changes in deployment requirements is needed. In one embodiment, developers can set up a scheduled task for development task comparison and set a comparison period, such as 1 hour, 2 hours, or 1 day. The start time of the first development task comparison period is the set specific time. After the first round of development task comparison is completed, the system waits for the next task comparison period. If the current time reaches the next task comparison period, the development task comparison task is triggered, and the specific task comparison process is executed, i.e., the steps in the above task comparison method embodiment.

[0083] In another embodiment, since online tasks and production scheduling tasks are usually in the form of Excel spreadsheets, developers can use the Visual Basic editor in Excel to create button controls in the editing area, embedding a task comparison function. This task comparison function includes the steps described in the task comparison method embodiment above, resulting in a task comparison button. When development tasks need to be compared, the operator only needs to click the pre-built task comparison button on the terminal's interface to trigger the development task comparison process, execute the steps described in the task comparison method embodiment above, and obtain the task comparison results. It is understood that in other embodiments, other methods can also be used to trigger the development task comparison process. In this embodiment, by providing both automatic and manual triggering methods, the needs of different scenarios can be met, improving the flexibility of task comparison processing. Furthermore, most machines have an Office environment deployed, meaning that task comparison using Excel VBA has a natural execution environment, making it more convenient than other language tools, and reducing the difficulty of migrating the solution program to different machines.

[0084] like Figure 3 As shown, in one embodiment, S600 includes: S620, extracting the online task identifier from the online development task dataset and the production scheduling task identifier from the production scheduling development task dataset, and storing the online task identifier and production scheduling task identifier data into data columns of different data tables to obtain the online task identifier column and the production scheduling task identifier data column.

[0085] S800 includes: S810, which performs a consistency comparison between the online task identifier column and the production scheduling task identifier data column to obtain the task comparison results.

[0086] In this embodiment, the task set can exist in the form of a list. Specifically, the online task identifier set includes an online task identifier column, and the production scheduling task identifier set includes a production scheduling task identifier column. The online task identifiers from the online development task dataset and the production scheduling task identifiers from the production scheduling development task dataset can be extracted according to the aforementioned field positions or delimiters. Then, the online task identifiers and production scheduling task identifiers are stored separately in data columns of different data tables. In this embodiment, the online task identifiers and production scheduling task identifiers can be imported into different Excel spreadsheets to obtain the online task identifier column and the production scheduling task identifier data column.

[0087] In this embodiment, by importing the online task identifier and the production scheduling task identifier into the data columns of different data tables, the data organization becomes clearer, and the results can be obtained faster when querying and comparing data, thereby improving data management efficiency and data quality, and facilitating subsequent data analysis and decision-making.

[0088] In one embodiment, the online task identifier and production scheduling task identifier data are stored in different data columns of different data tables, resulting in the online task identifier column and production scheduling task identifier data column, which include:

[0089] The first physical subsystem number and the online task number are concatenated and stored in a data column of a pre-built data table to obtain the online task identifier column.

[0090] The second physical subsystem number and the production scheduling task number are concatenated and stored in a data column of a pre-built data table to obtain the production scheduling task identifier data column.

[0091] The online task identifier includes the physical subsystem number to which the online task belongs and the online task number. The production scheduling task identifier data includes the physical subsystem number to which the production scheduling task belongs and the production scheduling task number. In this embodiment, the physical subsystem number to which the online task identifier belongs is taken as the first physical subsystem number, and the physical subsystem number to which the production scheduling task belongs is taken as the second physical subsystem number.

[0092] In practice, for each online task, the first physical subsystem number and the online task number can be concatenated, and the concatenated first physical subsystem number and online task number can be stored in a pre-built Excel data column to obtain the online task identifier column. For example, if the first physical subsystem number of the online task is A and the online task number is 0001, then concatenating the first physical subsystem number and the online task number will yield A_0001.

[0093] For each production scheduling task, the second physical subsystem number to which the task belongs and the production scheduling task number are concatenated. The concatenated second physical subsystem number and production scheduling task number are then stored in a pre-built Excel data column to obtain the production scheduling task identifier data column. For example, if the second physical subsystem number of the production scheduling task is B and the online task number is 0001, then the first physical subsystem number and the online task number are concatenated to obtain B_0001.

[0094] In this embodiment, by concatenating the physical subsystem number and the development task number, and storing the concatenated number as a data column, the data processing process can be simplified. It eliminates the need to process the physical subsystem number and development task number separately, and facilitates tracing back to the original number and related information. Furthermore, the concatenated number ensures data consistency, improves data integrity, and reduces the risk of data loss or duplication.

[0095] like Figure 4 As shown, in one embodiment, S800 includes:

[0096] S820: If an abnormal online task identifier is detected in the online task identifier set, the abnormal online task identifier is marked using the first identifier method. The abnormal online task identifier is development task identifier data that does not exist in the production scheduling task identifier set.

[0097] S840 If an abnormal production scheduling task identifier is detected in the production scheduling task identifier set, the abnormal production scheduling task identifier is marked using the second marking method. The abnormal production scheduling task identifier is the development task identifier data that does not exist in the online task identifier set.

[0098] In this embodiment, the process of comparing the consistency of the online task identifier set and the production scheduling task identifier set can be as follows: Traverse the online task identifier set, extract each online task identifier from the set, and match it with all production scheduling task identifiers in the production scheduling task identifier set. Check whether a production scheduling task identifier that matches or is identical to the online task identifier can be found in the production scheduling task identifier set. If no matching or identical production scheduling task identifier is found, the online task identifier is considered an abnormal task identifier, specifically a development task that needs to be canceled from production. In this case, it can be marked in red to facilitate the deletion of the development task by the operator in the production system. By matching each task identifier in the online task identifier set with all production scheduling task identifiers in the production scheduling identifier set in the above manner, the comparison result of the online task identifier set is obtained.

[0099] In practical applications, it's possible to iterate through the production scheduling task identifier set, extract each production scheduling task identifier, and match it against all deployment task identifiers in the deployment task identifier set. The process checks if a matching deployment task identifier can be found in the deployment task identifier set. If no matching deployment task identifier is found, it's considered an abnormal task identifier—specifically, a missing development task that needs to be added for deployment. In this case, it can be marked in green to facilitate operators adding the task in the production system.

[0100] In this embodiment, by using different marking methods to mark abnormal online tasks and abnormal production scheduling tasks, abnormal tasks can be accurately identified and classified, avoiding misjudgment or omission, and can also clearly instruct operators on how to handle different types of abnormal tasks, thus improving processing efficiency.

[0101] like Figure 5 As shown, in one embodiment, before S800, the method further includes: S700, performing data preprocessing on the online task identifier set and the production scheduling task identifier set, wherein the data preprocessing includes at least one of data deduplication, data format standardization and naming normalization.

[0102] S800 includes: S802, which performs a consistency comparison between the online task identifier set and the production scheduling task identifier set after data preprocessing to obtain the task comparison result.

[0103] In practical applications, due to the large number of development tasks and the numerous physical subsystems involved, there may be many-to-many relationships between development tasks and physical subsystems. Therefore, after obtaining the set of task identifiers for deployment and scheduling, data preprocessing is required. Specifically, data preprocessing may include removing duplicate data, handling null or missing values, and standardizing data formats and naming conventions.

[0104] Specifically, the naming standardization process can involve standardizing the naming conventions of task identifiers, such as task numbers, task IDs (identities), or task names, to ensure accurate matching during the comparison process. Data format standardization can involve unifying the data format of task numbers, task IDs, or task names, such as converting them to a uniform string or number format. After the above data preprocessing, a consistency comparison is further performed between the preprocessed set of online task identifiers and the set of production scheduling task identifiers to obtain the task comparison results.

[0105] In this embodiment, by preprocessing the online task identifier set and the production scheduling task identifier set, the standardization and quality of the data can be improved, which is conducive to improving the efficiency and accuracy of data comparison.

[0106] like Figure 6 As shown, in one embodiment, S802 includes: S812, sorting the pre-processed set of online task identifiers and the set of production task identifiers according to the online task identifier and the production task identifier, respectively, and performing a consistency comparison on the sorted set of online task identifiers and the set of production task identifiers to obtain the task comparison result.

[0107] Following the previous embodiment, after preprocessing the dataset, to facilitate data consistency comparison and improve the readability of task comparison results, the preprocessed online task identifier set and production scheduling task identifier set can be sorted. For example, the concatenated physical subsystem number and development task number can be used as the sorting criteria. The preprocessed online task identifier set and production scheduling task identifier set can be sorted in ascending or descending order, and then a consistency comparison can be performed on the sorted online task identifier set and production scheduling task identifier set. In this embodiment, sorting the online task identifier set and production scheduling task identifier set can improve the efficiency of subsequent consistency comparison operations and enhance the readability of task comparison results, making it easier for operators to promptly identify abnormal tasks.

[0108] like Figure 7 As shown, in one embodiment, after S800, the method further includes: S900, which calls a preset visualization tool to visualize the task comparison results, wherein the task comparison results include task priority, task dependency relationship and abnormal task identification results.

[0109] In practical applications, since the task comparison results involve a large number of tasks, the resulting task comparison results are usually quite numerous. To improve readability, the task comparison results can be organized and analyzed to extract task priorities, task dependencies, and abnormal task identifiers. This information should then be formatted into a data format suitable for visualization.

[0110] Specifically, preset visualization tools can be used to visualize the task comparison results. For example, based on task priority information, a task priority chart can be created using a selected visualization tool. This chart can use bar charts, pie charts, radar charts, etc., to display the distribution of task priorities. Tasks can also be color-coded or size-coded according to their priority for easy observation and comparison. Based on task dependency information, a task dependency graph can be created using visualization tools. This graph can use flowcharts, directed graphs, tree diagrams, etc., to display the dependencies between tasks. The connections between nodes and edges clearly show the sequence and dependencies between tasks. Based on the abnormal task identification results, an abnormal task identification chart can be created using visualization tools. This chart can use scatter plots, heatmaps, pie charts, etc., to display the distribution of abnormal tasks. Abnormal tasks can be identified using special markers or colors for quick identification and location. Finally, the drawn task priority chart, task dependency graph, and abnormal task identification chart are integrated to form a comprehensive visualization result. Different charts can be placed on the same canvas, or multiple charts can be created and grouped for display.

[0111] In this embodiment, the task comparison results are visualized by invoking a preset visualization tool. This allows for a clear and intuitive display of information such as task priority, task dependencies, and abnormal task identifiers, helping users better understand and analyze the task comparison results.

[0112] To provide a clearer explanation of the task comparison method provided in this application, a specific embodiment and accompanying drawings are described below. Figure 7 The specific embodiment is described below:

[0113] S200 determines in real time whether preset task comparison conditions can be triggered.

[0114] In one embodiment, developers can set up a scheduled task for comparing development tasks and set a comparison period, such as 1 hour, 2 hours, or 1 day. The start time of the first comparison period is the set specific time. After the first round of comparison is completed, the system waits for the next comparison period. If the current time reaches the next comparison period, the comparison is triggered, and the specific comparison process is executed, i.e., the steps in the above task comparison method embodiment. Alternatively, developers can use the Visual Basic editor in Excel to create a button control in the editing area, embedding a task comparison function. This function includes the steps in the above task comparison method embodiment, resulting in a task comparison button. When development tasks need to be compared, the operator only needs to click the pre-built task comparison button on the terminal's interface to trigger the comparison process, execute the steps in the above task comparison method embodiment, and obtain the comparison results.

[0115] S400, when the preset task comparison conditions are triggered, extracts the online development task dataset and the production scheduling development task dataset from different data sources respectively.

[0116] S620: Extract the online task identifier from the online development task dataset and the production scheduling task identifier from the production scheduling development task dataset, and store the online task identifier and production scheduling task identifier data into data columns of different data tables to obtain the online task identifier column and the production scheduling task identifier data column.

[0117] The online task identifier includes the physical subsystem number to which the online task belongs and the online task number. The production scheduling task identifier data includes the physical subsystem number to which the production scheduling task belongs and the production scheduling task number. In this embodiment, the physical subsystem number to which the online task identifier belongs is taken as the first physical subsystem number, and the physical subsystem number to which the production scheduling task belongs is taken as the second physical subsystem number.

[0118] In practice, for each online task, the first physical subsystem number and the online task number can be concatenated, and the concatenated first physical subsystem number and online task number can be stored in a pre-built Excel data column to obtain the online task identifier column. For example, if the first physical subsystem number of the online task is A and the online task number is 0001, then concatenating the first physical subsystem number and the online task number will yield A_0001.

[0119] For each production scheduling task, the second physical subsystem number to which the task belongs and the production task number are concatenated. The concatenated second physical subsystem number and production task number are then stored in a pre-built Excel data column to obtain the production task identifier data column. For example, if the second physical subsystem number of the production scheduling task is B and the online task number is 0001, then the first physical subsystem number and the online task number are concatenated to obtain B_0001.

[0120] S700 performs data preprocessing on the online task identifier set and the production scheduling task identifier set. Data preprocessing includes at least one of data deduplication, data format standardization, and naming standardization.

[0121] S812: Sort the pre-processed set of online task identifiers and the set of production task identifiers according to the online task identifier and the production task identifier, respectively. Then, perform a consistency comparison between the sorted set of online task identifiers and the set of production task identifiers to obtain the task comparison result.

[0122] S820: If an abnormal online task identifier is detected in the online task identifier set, the abnormal online task identifier is marked using the first identifier method. The abnormal online task identifier is development task identifier data that does not exist in the production scheduling task identifier set.

[0123] S840 If an abnormal production scheduling task identifier is detected in the production scheduling task identifier set, the abnormal production scheduling task identifier is marked using the second marking method. The abnormal production scheduling task identifier is the development task identifier data that does not exist in the online task identifier set.

[0124] The S900, a pre-set visualization tool, displays the task comparison results visually, including task priority, task dependencies, and abnormal task identification results.

[0125] The aforementioned data processing method links the previously isolated production and scheduling systems. By pre-setting task comparison conditions, it enables automatic comparison of development tasks. By extracting the online task identifier and the production scheduling task identifier, and comparing the sets of online task identifiers and production scheduling task identifiers, it can efficiently and accurately compare whether the task identifiers in the two systems are consistent, quickly detecting any missing or duplicate task identifiers in the production and scheduling systems, thus improving the efficiency of development task comparison. Furthermore, this approach also allows operators to promptly identify abnormal tasks and add or delete online task scopes, helping to ensure the consistency between the online task scope and the production task scope, and improving data security in the production environment.

[0126] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0127] Based on the same inventive concept, this application also provides a data processing apparatus for implementing the data processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more data processing apparatus embodiments provided below can be found in the limitations of the data processing method described above, and will not be repeated here.

[0128] In one embodiment, such as Figure 8 As shown, a data processing device 800 is provided, including: a data extraction module 810, a data interception and processing module 820, and a data comparison module 830, wherein:

[0129] The data extraction module 810 is used to extract the online development task dataset and the production scheduling development task dataset from different data sources when the preset task comparison conditions are triggered.

[0130] The data capture and processing module 820 is used to capture the online task identifier in the online development task dataset and the production scheduling task identifier in the production scheduling development task dataset to obtain the online task identifier set and the production scheduling task identifier set.

[0131] The data comparison module 830 is used to perform a consistency comparison between the online task identifier set and the production scheduling task identifier set to obtain the task comparison results.

[0132] In the aforementioned data processing device, when preset task comparison conditions are triggered, the online development task dataset and the production scheduling development task dataset are extracted from different data sources. Then, the online task identifiers from the online development task dataset and the production scheduling task identifiers from the production scheduling development task dataset are extracted to obtain the online task identifier set and the production scheduling task identifier set. Finally, the consistency of the online task identifier set and the production scheduling task identifier set is compared to obtain the task comparison result. The entire solution connects the originally isolated production system and the production scheduling system. By pre-setting task comparison conditions, automatic comparison of development tasks can be achieved. By extracting the online task identifiers and production scheduling task identifiers and comparing the online task identifier set and the production scheduling task identifier set, the consistency between online tasks and production tasks can be compared efficiently and accurately. Missing or duplicate task identifiers in online tasks and production tasks can be quickly detected, improving the efficiency of development task comparison. Furthermore, adopting the above solution also makes it easier for operators to promptly identify abnormal tasks and add or delete tasks in the online task scope, which helps to ensure the consistency between the online task scope and the production scheduling task scope, and improves the data security of the production environment.

[0133] In one embodiment, the online task identifier includes the first physical subsystem number and the online task number, and the production scheduling task identifier data includes the second physical subsystem number and the production scheduling task number.

[0134] The data capture and processing module 820 is also used to concatenate the first physical subsystem number and the online task number, and store the concatenated first physical subsystem number and online task number into a data column of a pre-built data table to obtain an online task identifier column; and to concatenate the second physical subsystem number and the production scheduling task number, and store the concatenated second physical subsystem number and the production scheduling task number into a data column of a pre-built data table to obtain a production scheduling task identifier data column.

[0135] In one embodiment, the data comparison module 830 is further configured to, if an abnormal online task identifier is detected in the online task identifier set, mark the abnormal online task identifier using a first marking method, wherein the abnormal online task identifier is development task identifier data that does not exist in the production scheduling task identifier set; and if an abnormal production scheduling task identifier is detected in the production scheduling task identifier set, mark the abnormal production scheduling task identifier using a second marking method, wherein the abnormal production scheduling task identifier is development task identifier data that does not exist in the online task identifier set.

[0136] like Figure 9 As shown, in one embodiment, the apparatus further includes a data preprocessing module 822, used to perform data preprocessing on the online task identifier set and the production scheduling task identifier set, wherein the data preprocessing includes at least one of data deduplication, data format standardization and naming normalization.

[0137] The data comparison module 830 is also used to perform consistency comparison between the online task identifier set and the production scheduling task identifier set after data preprocessing.

[0138] In one embodiment, the data comparison module 830 is further configured to sort the pre-processed set of online task identifiers and the set of production task identifiers according to the online task identifier and the production task identifier, respectively, and perform a consistency comparison on the sorted set of online task identifiers and the set of production task identifiers to obtain the task comparison result.

[0139] like Figure 9 As shown, in one embodiment, the device further includes a visualization module 840, which is used to call a preset visualization tool to visualize the task comparison results. The task comparison results include task priority, task dependency relationship and abnormal task identification results.

[0140] Each module in the aforementioned data processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0141] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 10 As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores data such as online development task datasets, production scheduling development task datasets, and task comparison results. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a data processing method.

[0142] Those skilled in the art will understand that Figure 10The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0143] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described data processing method embodiments.

[0144] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above-described data processing method embodiments.

[0145] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described data processing method embodiments.

[0146] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0147] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0148] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A data processing method, characterized in that, The method includes: When the preset task comparison conditions are triggered, the online development task dataset and the production scheduling development task dataset are extracted from different data sources respectively. Extract the online task identifier from the online development task dataset and the production scheduling task identifier from the production scheduling development task dataset. The online task identifier includes the first physical subsystem number and the online task number, and the production scheduling task identifier data includes the second physical subsystem number and the production scheduling task number. The first physical subsystem number and the online task number are concatenated, and the concatenated first physical subsystem number and the online task number are stored in the data column of a pre-built data table to obtain the online task identifier column; The second physical subsystem number and the production scheduling task number are concatenated, and the concatenated second physical subsystem number and the production scheduling task number are stored in the data column of a pre-built data table to obtain the production scheduling task identifier data column; The consistency comparison between the online task identifier set and the production scheduling task identifier set is performed to obtain the task comparison result.

2. The method according to claim 1, characterized in that, The consistency comparison between the online task identifier set and the production scheduling task identifier set includes: If an abnormal online task identifier is detected in the online task identifier set, the abnormal online task identifier is marked using the first marking method. The abnormal online task identifier is development task identifier data that does not exist in the production scheduling task identifier set. If an abnormal production scheduling task identifier is detected in the production scheduling task identifier set, the abnormal production scheduling task identifier is marked using a second marking method. The abnormal production scheduling task identifier is development task identifier data that does not exist in the online task identifier set.

3. The method according to claim 1, characterized in that, Before performing a consistency comparison between the online task identifier set and the production scheduling task identifier set, the process also includes: Data preprocessing is performed on the online task identifier set and the production scheduling task identifier set. Data preprocessing includes at least one of data deduplication, data format standardization, and naming standardization. A consistency comparison is performed between the online task identifier set and the production scheduling task identifier set to obtain the task comparison results, including: The online task identifier set and the production scheduling task identifier set after data preprocessing are compared for consistency to obtain the task comparison result.

4. The method according to claim 3, characterized in that, The consistency comparison between the preprocessed online task identifier set and the production scheduling task identifier set yields the following task comparison results: Sort the set of online task identifiers and the set of production task identifiers after data preprocessing according to the online task identifier and the production task identifier; The consistency comparison between the sorted online task identifier set and the production scheduling task identifier set is performed to obtain the task comparison results.

5. The method according to any one of claims 1 to 4, characterized in that, The preset task comparison conditions include any one of the following: The first requirement is to reach the preset development task comparison cycle; The second item is receiving a click operation on the task comparison button, which is built based on the Visual Basic editor of Excel.

6. The method according to any one of claims 1 to 4, characterized in that, After obtaining the task comparison results, the process also includes: The preset visualization tool is invoked to visualize the comparison results of the task. The task comparison results include task priority, task dependencies, and abnormal task identification results.

7. A data processing apparatus, characterized in that, The device includes: The data extraction module is used to extract the online development task dataset and the production scheduling development task dataset from different data sources when the preset task comparison conditions are triggered. The data extraction and processing module is used to extract the online task identifier from the online development task dataset and the production scheduling task identifier from the production scheduling development task dataset. The online task identifier includes a first physical subsystem number and an online task number, and the production scheduling task identifier data includes a second physical subsystem number and a production scheduling task number. The module concatenates the first physical subsystem number and the online task number, and stores the concatenated first physical subsystem number and the online task number into a data column of a pre-built data table to obtain an online task identifier column. The module also concatenates the second physical subsystem number and the production scheduling task number, and stores the concatenated second physical subsystem number and the production scheduling task number into a data column of a pre-built data table to obtain a production scheduling task identifier data column. The data comparison module is used to perform a consistency comparison between the online task identifier set and the production scheduling task identifier set to obtain the task comparison result.

8. The apparatus according to claim 7, characterized in that, The data comparison module is further configured to, if an abnormal online task identifier is detected in the online task identifier set, mark the abnormal online task identifier using a first marking method, wherein the abnormal online task identifier is development task identifier data that does not exist in the production scheduling task identifier set; and if an abnormal production scheduling task identifier is detected in the production scheduling task identifier set, mark the abnormal production scheduling task identifier using a second marking method, wherein the abnormal production scheduling task identifier is development task identifier data that does not exist in the online task identifier set.

9. The apparatus according to claim 7, characterized in that, The device further includes a data preprocessing module for preprocessing the online task identifier set and the production scheduling task identifier set. The data preprocessing includes at least one of data deduplication, data format standardization, and naming standardization. The data comparison module is also used to perform a consistency comparison between the online task identifier set and the production scheduling task identifier set after data preprocessing.

10. The apparatus according to claim 9, characterized in that, The data comparison module is further configured to sort the pre-processed set of online task identifiers and the set of production task identifiers according to the online task identifier and the production task identifier, respectively, and perform a consistency comparison on the sorted set of online task identifiers and the set of production task identifiers to obtain the task comparison result.

11. The apparatus according to any one of claims 7 to 10, characterized in that, The device also includes a visualization module for calling a preset visualization tool to visualize the task comparison results, wherein the task comparison results include task priority, task dependency relationship and abnormal task identification results.

12. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

14. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images