Data processing method, storage medium and product

Abstract

The invention discloses a data processing method, a storage medium and a product. The method comprises the steps of reading first task data needing to be processed by a data processing task from a data source; detecting the structure of the first task data to obtain a detection result; if the detection result indicates that the structure of the first task data is changed, a first class in the data processing task is triggered to be initialized, and the first class comprises a data writing function; and writing the first task data into a corresponding position of the data source through the first class in the initialized data processing task. The data loss probability and the processing delay phenomenon can be reduced.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method, storage medium and product. Background Art

[0002] Apache Flink is an open-source real-time computing and processing engine for big data. Users can create tasks in Apache Flink, consume data from data sources through tasks, and write the consumed data to corresponding locations.

[0003] However, when writing data consumed by Apache Flink from the data source to the corresponding location, some factors, such as the complexity and diversity of the data, may cause the write to fail, requiring the task in Apache Flink to be restarted, resulting in data loss and processing delays. Summary of the Invention

[0004] In response to the above technical problems, the embodiments of the present application provide a data processing method, storage medium and product, which can reduce the probability of data loss and reduce processing delays.

[0005] In a first aspect, an embodiment of the present application provides a data processing method, comprising:

[0006] Reading first task data to be processed by the data processing task from a data source;

[0007] Detecting the structure of the first task data to obtain a detection result;

[0008] If the detection result indicates that the structure of the first task data has changed, the first type of the data processing task is triggered to be initialized, and the first type includes a data writing function;

[0009] The first task data is written into the corresponding position of the data source through the first type of initialized data processing tasks.

[0010] In a second aspect, an embodiment of the present application provides a data processing device, including:

[0011] A reading module, configured to read first task data to be processed by a data processing task from a data source;

[0012] A detection module, configured to detect the structure of the first task data and obtain a detection result;

[0013] a trigger module configured to trigger initialization of a first type of the data processing tasks if the detection result indicates that the structure of the first task data has changed, wherein the first type includes a data writing function;

[0014] The writing module is used to write the first task data into the corresponding position of the data source through the first type of the initialized data processing task.

[0015] In a third aspect, an embodiment of the present application further provides a computer device, comprising a memory storing a plurality of instructions; a processor loading instructions from the memory to execute the steps of any data processing method provided in the embodiment of the present application.

[0016] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores a plurality of instructions suitable for loading by a processor to execute the steps of any data processing method provided in an embodiment of the present application.

[0017] In a fifth aspect, an embodiment of the present application further provides a computer program product, comprising a computer program or instructions, which, when executed by a processor, implements the steps of any data processing method provided in the embodiment of the present application.

[0018] In an embodiment of the present application, first task data to be processed by a data processing task is read from a data source, and the structure of the first task data is detected to obtain a detection result; if the detection result indicates that the structure of the first task data has changed, the first category in the data processing task is triggered to be initialized, and the first category includes a data writing function; through the first category in the initialized data processing task, the first task data is written to the corresponding position of the data source, so that when the structure of the first task data changes, there is no need to restart the data processing task, and the first task data can also be successfully written to the corresponding position of the data source, thereby reducing the probability of data loss and reducing the occurrence of processing delays. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 Schematic diagram of the application environment of the data processing method provided in the embodiment of the present application;

[0021] Figure 2 is another schematic diagram of an application environment of the data processing method provided in an embodiment of the present application;

[0022] Figure 3 This is a flow chart of an embodiment of the data processing method provided in the embodiments of the present application;

[0023] Figure 4 is a schematic diagram of a data processing task provided in an embodiment of the present application;

[0024] Figure 5 is another schematic diagram of a data processing task provided in an embodiment of the present application;

[0025] Figure 6 Schematic diagram of the system framework of the data processing method provided in the embodiment of the present application;

[0026] Figure 7 is another schematic diagram of the system framework of the data processing method provided in the embodiments of the present application;

[0027] Figure 8 This is a flow chart of another embodiment of the data processing method provided in the embodiments of the present application;

[0028] Figure 9 is a structural diagram of a data processing device provided in an embodiment of the present application;

[0029] Figure 10 It is a schematic diagram of the internal structure of the computer device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0030] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of this application. At the same time, in the description of the embodiments of the present application, the terms "first", "second", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more features. In the description of the embodiments of the present application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0031] In one embodiment of the present application, the data processing method may be run on a local terminal device or a server.

[0032] In order to better understand the data processing method, storage medium and program product provided by the embodiments of the present application, the application environment applicable to the embodiments of the present application is described below.

[0033] See also Figure 1 , Figure 1Schematic diagram of an application environment of the data processing method provided by an embodiment of the present application is shown. As an implementation method, the data processing method provided by the embodiment of the present application can be applied to a computer device. The computer device can be Figure 1 The server 110 shown in FIG. 1 can be connected to the terminal device 120 via a network. The network is used to provide a medium for a communication link between the server 110 and the terminal device 120. The network can include various connection types, such as wired communication links, wireless communication links, etc., which are not limited in the embodiments of the present application. Alternatively, in other embodiments, the computer device can also be a smartphone, a laptop computer, etc.

[0034] It should be understood that Figure 1 The server 110, network, and terminal device 120 are merely illustrative. Any number of servers, networks, and terminal devices may be provided as needed. For example, the server 110 may be a physical server or a server cluster consisting of multiple servers, and the terminal device 120 may be a mobile phone, tablet, desktop computer, laptop computer, or the like. It will be appreciated that embodiments of the present application may also allow multiple terminal devices 120 to access the server 110 simultaneously.

[0035] As another embodiment, the application environment of the data processing method of the embodiment of the present application can be as follows: Figure 2 As shown. Among them, the computer device of the embodiment of the present application can be Figure 2 In the server 201, a big data engine 2011 can be deployed on the server 201. The big data engine can be, for example, Apache Flink. The big data engine can include data processing tasks. The server 201 reads the first task data from the data source 202 through the data processing tasks and writes the first task data to the data lake 203.

[0036] It should be understood that Figure 2 The server 201 in the example may be a physical server or a server cluster composed of multiple servers.

[0037] The following is a detailed description of each step in conjunction with the accompanying drawings. In this embodiment, a computer device is used as an example. It should be noted that the order in which the following embodiments are described does not limit the preferred order of the embodiments. Although the flowcharts illustrate a logical order, in some cases, the steps shown or described may be performed in a different order than that shown in the accompanying drawings.

[0038] In related technologies, Apache Flink is an open source real-time computing and processing big data engine. Users can deploy Apache Flink components on the server, deploy data processing tasks in the Apache Flink components through the hudi-flink-bundle tool, and run data processing tasks to consume data from the data source and write the consumed data to the data lake.

[0039] Before running a data processing task, you need to set the data structure first. If the data structure changes during the running of the data processing task, data writing will fail, requiring the data processing task to be restarted, resulting in data loss and processing delays.

[0040] In addition, the hudi-flink-bundle tool has requirements for the versions of Apache Flink components and the data lake. If the versions of Apache Flink components and the data lake do not meet the requirements of the hudi-flink-bundle tool, the hudi-flink-bundle tool cannot write data to the data lake.

[0041] In an embodiment of the present application, the server where the big data engine is located reads the first task data to be processed by the data processing task from the data source, detects the structure of the first task data, and obtains a detection result; if the detection result indicates that the structure of the first task data has changed, the first category in the data processing task is triggered to be initialized, and the first category includes a data writing function; through the first category in the initialized data processing task, the first task data is written to the corresponding position of the data source, and the first task data is written according to the detection result indicating that the structure has changed or the detection result indicating that the structure has not changed, so that the first task data can be successfully written into the data lake, so that when the structure of the data changes, there is no need to restart the data processing task, and the first task data can also be successfully written into the data lake, reducing the probability of data loss and reducing processing delays. In addition, there are no requirements for the version of the big data engine and the version of the data lake, reducing the dependence on the version of the big data engine and the version of the data lake.

[0042] Please refer to Figure 3 The specific process of the data processing method may be as follows: Steps 301 to 304, wherein:

[0043] Step 301: read first task data to be processed by a data processing task from a data source.

[0044] The data processing task refers to a workflow that reads data from a data source and writes data to it. The data source refers to where the data for the first task is generated. The type of data source can be set based on actual circumstances. For example, the data source can be a database or a message queue, and this embodiment of the application does not limit this.

[0045] The first task data refers to the data in the data source. The data in the data source can come from the terminal device, or the data in the data source can come from a third-party service platform. In other words, the first task data can come from the terminal device or the third-party service platform, which is not limited in this embodiment of the present application.

[0046] The type of the first task data can be set according to actual conditions. For example, the first task data can be at least one of text data, image data, and video data.

[0047] In some embodiments, the computer device may include a big data engine, which creates a data processing task and a data source corresponding to the data processing task through the big data engine, and runs the data processing task so as to read the first task data from the data source through the data processing task.

[0048] Among them, the type of big data engine can be set according to actual conditions. For example, the big data engine can be Apache Spark or Apache Flink, which is not limited in this embodiment of the present application.

[0049] In some embodiments, the data processing task includes at least one task operator, so that the data processing method provided by this embodiment is executed through at least one task operator. The task operator refers to a function or method used to process the first task data.

[0050] Step 302: Detect the structure of the first task data to obtain a detection result.

[0051] The structure (Schema) of the first task data refers to the composition of the first task data, which defines the fields, data types, and the relationship between the fields, so that the first task data has consistency and parsability, so as to facilitate querying, writing, and analyzing the first task data.

[0052] The structure of the first task data can be set according to actual needs. For example, if the first task data is "Name: Zhang San, Age: 10, Score: 100 points", the structure of the first task data includes three fields, namely name, age and score.

[0053] The method of detecting the structure of the first task data and obtaining the detection result can be set according to actual conditions, and the embodiment of the present application does not limit it here.

[0054] For example, in some embodiments, the process of detecting the structure of the first task data and obtaining the detection result may be:

[0055] Get the stored target data structure;

[0056] Comparing the target data structure with the structure of the first task data;

[0057] If the target data structure is different from the structure of the first task data, a detection result indicating that a structure change has occurred is obtained;

[0058] If the target data structure is identical to the structure of the first task data, a detection result indicating that the structure has not changed is obtained.

[0059] The required data structure is set in the data processing task. When the data structure set in the data processing task is the same as the structure of the first task data, the first task data can be written to the corresponding location through the data processing task. When the data structure set in the data processing task is different from the structure of the first task data, the first task data cannot be written to the corresponding location through the data processing task. In this embodiment, the data structure set in the data processing task is stored as a target data structure, so that the target data structure can be used to determine whether the structure of the first task data is the same as the data structure in the data processing task.

[0060] In an embodiment of the present application, a stored target data structure is obtained, and the target data structure and the structure of the first task data are compared. If the target data structure and the structure of the first task data are different, a detection result indicating that the structure has changed is obtained. If the target data structure and the structure of the first task data are the same, a detection result indicating that the structure has not changed is obtained, thereby realizing the judgment of whether the structure of the first task data has changed through the pre-stored target data.

[0061] In some embodiments, the process of detecting the structure of the first task data and obtaining the detection result may also be:

[0062] Get the structure of historical task data;

[0063] If the structure of the historical task data is the same as the structure of the first task data, a detection result indicating that the structure has not changed is obtained;

[0064] If the structure of the historical task data is different from the structure of the first task data, a detection result indicating that a structure change occurs is obtained.

[0065] The historical task data refers to the data that the computer device reads from the data source and writes to the corresponding location before reading the first task data. Alternatively, the historical task data may refer to the data that was last read from the data source and written to the corresponding location before reading the first task data.

[0066] Since historical task data can be written to corresponding locations through data processing tasks, the structure of the historical task data is the same as the data structure in the data processing task. When the structure of the first task data is the same as the structure of the historical task data, it means that the structure of the first task data is the same as the data structure in the data processing task. When the structure of the first task data is different from the structure of the historical task data, it means that the structure of the first task data is different from the data structure in the data processing task.

[0067] In an embodiment of the present application, the structure of the historical task data is obtained. If the structure of the historical task data is the same as the structure of the first task data, a detection result indicating that the structure has not changed is obtained. If the structure of the historical task data is different from the structure of the first task data, a detection result indicating that the structure has changed is obtained, thereby realizing the judgment of whether the structure of the first task data has changed through the structure of the historical task data.

[0068] It is understandable that when the first task data is the data read from the data source for the first time, there is no historical task data, and thus no data structure of the historical task data. In this case, it can be assumed that the structure of the first task data has changed.

[0069] Step 303: If the detection result indicates that the structure of the first task data has changed, the first type of data processing tasks is triggered to be initialized, and the first type includes a data writing function.

[0070] The first category refers to methods within a data processing task that have a write function. The data structure configured within the data processing task is configured within the first category of the data processing task. If the structure of the first task data changes, it indicates that the structure of the first task data is different from the data structure configured within the first category. If the structure of the first task data remains unchanged, it indicates that the structure of the first task data is the same as the data structure configured within the first category.

[0071] Triggering the initialization of the first class in a data processing task can be understood as the process of initializing the data structure configured in the first class and creating a target instance based on the initialized data structure. Because the initialized data structure can be the same as the structure of the first task data, the data structure of the target instance created based on the initialized data structure is the same as the structure of the first task data.

[0072] Therefore, when the detection result indicates that the structure has not changed, it means that the data structure set in the first category of the data processing task is the same as the structure of the first task data, and the first task data can be written to the corresponding position through data task processing. When the detection result indicates that the structure has changed, it means that the data structure set in the first category of the data processing task is different from the structure of the first task data. At this time, the first task data cannot be written to the corresponding position through data task processing. Therefore, the first category in the data processing task can be triggered to be initialized, so that the first task data can be written to the corresponding position subsequently through the target instance of the first category in the initialized data processing task.

[0073] Step 304 : writing the first task data into the corresponding position of the data source through the first type of the initialized data processing task.

[0074] Among them, the corresponding location of the data source can refer to the final location where the data in the data source is stored. For example, the corresponding location of the data source can be the data lake of the data source, or the corresponding location of the data source can be the intermediate location where the data in the data source is stored. For example, the corresponding location can be a cache. In this case, the first task data is first stored in the cache, and then the first task data in the cache is stored in the data lake.

[0075] In an embodiment of the present application, when the detection result indicates that the structure has not changed, the first task data can be directly written to the corresponding position through the data processing task. When the detection result indicates that the structure has changed, the first category in the data processing task can be initialized based on the structure of the first task data, so that the first task data can be written to the corresponding position through the first category in the initialized data processing task, thereby ensuring the successful writing of the first task data. When the structure of the data changes, there is no need to manually restart the data processing task, and the first task data can be successfully written to the corresponding position, thereby reducing the probability of data loss and reducing processing delays.

[0076] In some embodiments, triggering the initialization of the first type of data processing tasks includes:

[0077] Based on the structure of the first task data, updating the data structure configured in the first category of data processing tasks;

[0078] Create a target instance in the updated first class, where the data structure configured in the target instance is the same as the structure of the first task data;

[0079] Writing the first task data to the corresponding location of the data source through the first type of initialized data processing task includes:

[0080] The first task data is written to the corresponding position of the data source through the target instance of the first type of the initialized data processing task.

[0081] After creating the target instance in the updated first class, the initialized data processing task can be obtained.

[0082] Specifically, the first class may include a constructor, and the data structure configured in the first class can be defined by the properties of the constructor in the first class. At this time, triggering the initialization of the first class in the data processing task may refer to the process of initializing the properties of the constructor of the first class in the data processing task and creating a target instance based on the initialized constructor. Since the properties of the constructor in the first class are used to define the data structure, after initializing the properties of the constructor, the data structure in the initialized constructor can be made the same as the structure of the first task data, so that the data structure of the target instance created based on the initialized constructor can be made the same as the data structure in the initialized constructor, and further the data structure of the target instance can be made the same as the structure of the first task data, so that the first task data can be written to the corresponding position of the data source through the target instance of the first class of the initialized data processing task.

[0083] Among them, the method of initializing the constructor in the first category can be set according to actual conditions. For example, the constructor in the first category can be modified or a new constructor can be created in the first category to initialize the constructor in the first category. This embodiment does not limit this.

[0084] In this embodiment, based on the structure of the first task data, the data structure configured in the first category of the data processing task is updated, and a target instance is created in the updated first category. The data structure configured in the target instance is the same as the structure of the first task data. The first task data is written to the corresponding position of the data source through the initialized target instance of the first category of the data processing task, thereby updating the data structure configured in the first category of the data processing task so that the first task data can be written through the updated target instance in the first category, ensuring that the data structure configured in the target instance is the same as the structure of the first task data, and further improving the probability of successfully writing the first task data.

[0085] In some embodiments, when initializing the constructor in the first class by modifying the existing constructor in the first class, updating the data structure configured in the first class of data processing tasks based on the structure of the first task data includes:

[0086] Based on the structure of the first task data, the constructor in the first class of the data processing task is modified so that the data structure configured in the modified constructor is the same as the structure of the first task data;

[0087] Create the target instance in the updated first class, including:

[0088] Create a target instance based on the modified constructor in the updated first class.

[0089] Among them, the modification of the constructor in the first category of data processing tasks may refer to modifying the properties of the constructor in the first category of data processing tasks so that the data structure in the properties of the constructor in the updated first category is the same as the structure of the first task data, and thus the data structure in the target instance created based on the modified constructor in the updated first category is the same as the structure of the first task data.

[0090] In this embodiment, based on the structure of the first task data, the constructor in the first class of the data processing task is modified to achieve the update of the data structure configured in the first class of the data processing task. The data structure configured in the modified constructor is the same as the structure of the first task data. Based on the modified constructor in the updated first class, the target instance is created without creating a new constructor, so as to ensure the simplicity of the code in the data processing task and reduce code redundancy.

[0091] In some embodiments, when initializing the constructor in the first class by creating a new constructor in the first class, updating the data structure configured in the first class of the data processing task based on the structure of the first task data includes:

[0092] Based on the structure of the first task data, a new constructor is created in the first class of the data processing task to update the data structure configured in the first class of the data processing task, where the data structure configured in the new constructor is the same as the structure of the first task data;

[0093] At this point, create the target instance in the updated first class, including:

[0094] Create a target instance based on the new constructor in the updated first class.

[0095] Wherein, based on the structure of the first task data, after a new constructor is created in the first type of data processing task, the data structure configured in the new constructor can be made the same as the structure of the first task data.

[0096] Specifically, when the difference between the structure of the first task data and the data structure in the first class of data processing tasks refers to the structure of the first task data, when new fields are added relative to the data structure in the first class of data processing tasks, a new constructor can be created by inheriting the first class of data processing tasks so as to reuse the existing constructor in the first class of data processing tasks so as to reuse the code.

[0097] It is understood that before creating a new constructor in the first category of data processing tasks, the first category of data processing tasks includes at least one existing constructor. In the process of creating a new constructor in the first category of data processing tasks based on the structure of the first task data, the existing constructor in the first category of data processing tasks may be deleted or retained, and this embodiment does not impose any limitation on this.

[0098] In this embodiment, based on the structure of the first task data, a new constructor is created in the first class of the data processing task to update the data structure configured in the first class of the data processing task. The data structure configured in the new constructor is the same as the structure of the first task data, so that the data structure in the target instance created based on the new constructor in the updated first class is the same as the structure of the first task data, so as to improve the readability, flexibility and extensibility of the code of the data processing task.

[0099] In some embodiments, while retaining the existing constructor in the first class of the data processing task, creating a new constructor in the first class of the data processing task based on the structure of the first task data includes:

[0100] Determine whether the existing constructors in the first category of data processing tasks include a constructor of the structure of the first task data;

[0101] If it does not exist, a new constructor is created in the first class in the data processing task based on the structure of the first task data.

[0102] At this time, this embodiment also includes:

[0103] If there is a constructor including a structure of the first task data among existing constructors in the first class of the data processing task, a target instance is created based on the constructor including the structure of the first task data.

[0104] In this embodiment, the existing constructors in the first category of data processing tasks are retained, and it is determined whether the existing constructors in the first category of data processing tasks include the constructor of the structure of the first task data. If not, a new constructor is created in the first category of the data processing task based on the structure of the first task data. If the existing constructors in the first category of data processing tasks include a constructor of the structure of the first task data, a target instance is created based on the constructor including the structure of the first task data, so that there is no need to frequently create new constructors, and the existing constructors in the first category of data processing tasks can be reused.

[0105] In some embodiments, the corresponding location is a cache space, and before updating the data structure configured in the first category of data processing tasks based on the structure of the first task data, the method further includes:

[0106] The second task data stored in the cache space is written to the data lake through the first type target instance in the data processing task.

[0107] After the data in the data source is stored in the cache space, the data stored in the cache space by the data source can be referred to as second task data. The second task data can be historical task data obtained from the data source before the first task data is stored in the cache space.

[0108] Because the structure of the second task data in the cache space is the same as the data structure configured in the target instance of the first type in the data processing task, the second task data in the cache space cannot be written to the data lake through the initialized target instance of the first type of the data processing task. Therefore, before initializing the first type in the data processing task, the second task data in the cache space is first written to the data lake through the target instance of the first type in the data processing task.

[0109] In a data lake, data is stored in the Parquet format, which has strong compatibility with the data structure. However, in related technologies, when the data structure changes, the data processing task needs to be restarted, resulting in the data structure being rarely changed. As a result, the data lake's compatibility with the data structure is not fully utilized.

[0110] In this embodiment, if the detection result indicates that the structure has changed, the second task data in the cache space is written into the data lake through the target instance of the first category in the data processing task, triggering the initialization of the first category in the data processing task. The first task data is written into the cache space through the target instance of the first category in the initialized data processing task, so that the structure of the data can be changed in real time as needed, so that the compatibility of the data lake with the data structure can be fully utilized.

[0111] In some embodiments, in the process of writing the second task data in the cache space into the data lake, a parquet file can be created to store the second task data in the cache space through the parquet file, or the second task data in the cache space can be stored in a parquet file already in the data lake.

[0112] In some embodiments, a data processing task includes a first task operator and a second task operator, the first task operator is used to detect the structure of the first task data, and the second task operator includes a first class. If the detection result indicates that the structure of the first task data has changed, the first class in the data processing task is triggered to be initialized, including:

[0113] If the detection result indicates that the structure of the first task data has changed, a structure change event is generated by the first task operator, and the structure change event is transmitted to the second task operator;

[0114] When a structure change event is received through the second task operator, the first class is triggered to be initialized.

[0115] The first task operator refers to a function or method for detecting the structure of the first task data. The second task operator refers to a function or method for writing the first task data to the corresponding location. When the data processing task includes the first task operator and the second task operator, the data processing task can be, for example, Figure 4 shown.

[0116] In this embodiment, if the detection result indicates that the structure of the first task data has changed, a structure change event (SchemaEvent) is generated by the first task operator, and the structure change event is passed to the second task operator. When the structure change event is received by the second task operator, the first class is triggered for initialization, and a structure change event is set between the first task operator and the second task operator, so that the structure of the first task data is passed through the structure change event, so that the second task operator can be initialized according to the structure of the first task data in the structure change event.

[0117] Optionally, after a structure change event is generated by the first task operator, a data event (DataEvent) can be generated based on the first task data, and the data structure can be passed to the second task operator, so that after the data event is received by the initialized second task operator, the first task data can be extracted from the data event from the first class in the initialized second task operator, and the first task data can be written to the corresponding location.

[0118] In some embodiments, when writing the first task data to a corresponding location, it is necessary to perform calculations on the first task data to obtain a partition of the first task data at the corresponding location, and then write the first task data to the partition at the corresponding location. Therefore, in some embodiments, data task processing may include a first task operator, a second task operator, and a third task operator. The first task operator is used to detect the structure of the first task data, the second task operator is used to perform calculations on the first task data to obtain a partition identifier, and the third task operator is used to write the first task data to the partition corresponding to the partition identifier at the corresponding location.

[0119] It can be understood that when the data processing task includes a first task operator, a second task operator and a third task operator, the first category may include a first subclass and a second subclass, and the first subclass and the second subclass jointly implement the data writing function, triggering the initialization of the first category in the data processing task, which may refer to initializing the first subclass in the second task operator and initializing the second subclass in the third task operator.

[0120] Specifically, the structure of the first task data can be detected by the structural component (SchemaCompator) in the first task operator. The first subclass can be called the key extractor (KeyExtractor) in the second task operator. At this time, the partition of the first task data can be calculated by the key extractor in the second task operator. The second subclass can be called the writer (Hudi Writer) in the third task operator. The first task data is written to the partition at the corresponding position by the writer in the third task operator. At this time, the data processing task can be as follows Figure 5 shown.

[0121] In some embodiments, if the detection result indicates that the structure of the first task data has changed, generating a structure change event through the first task operator includes:

[0122] If the detection result indicates that the structure of the first task data has changed, a structure change request is sent to the management center through the first task operator;

[0123] Receive verification information returned by the management center based on the structure change request;

[0124] In the case where the verification information indicates a structural change, a structural change event is generated by the first task operator.

[0125] The SourceCoordinator is used to manage and schedule data processing tasks in the big data engine. When the big data engine is Apache Flink, it can be deployed in the Flink JobManager.

[0126] The structure change request includes the structure of the first task data. After receiving the structure change request, the management center can determine whether the structure of the first task data is the required structure. If the structure of the first task data is the required structure, the management center returns verification information indicating that the structure has been changed. If the structure of the first task data is not the required structure, the management center returns verification information indicating that the structure has not been changed.

[0127] Specifically, before changing the structure of the data in the data source, the user may first store the structure of the first task data in the management center. After receiving the structure change request, if the structure stored in the management center is the same as the structure of the first task data in the structure change request, the management center determines that the structure of the first task data in the structure change request is the required structure. If the structure stored in the management center is different from the structure of the first task data in the structure change request, the management center determines that the structure of the first task data in the structure change request is not the required structure.

[0128] Alternatively, before changing the structure of the data in the data source, the user may first store the structure of the first task data in the structure configuration center (Schema Config Server). After receiving the structure change request, the management center forwards the structure change request to the structure configuration center. If the structure stored in the structure configuration center is the same as the structure of the first task data in the structure change request, it is determined that the structure of the first task data in the structure change request is the required structure, and verification information indicating the structure change is returned to the management center. If the structure stored in the structure configuration center is different from the structure of the first task data in the structure change request, it is determined that the structure of the first task data in the structure change request is not the required structure, and verification information indicating that the structure is not changed is returned to the management center.

[0129] When the structure configuration center confirms whether the structure of the first task data is the required structure, the system framework diagram of the data processing method of this embodiment can be as follows: Figure 6 shown.

[0130] In this embodiment, if the detection result indicates that the structure of the first task data has changed, a structure change request is sent to the management center through the first task operator, and verification information returned by the management center based on the structure change request is received. When the verification information indicates a structure change, a structure change event is generated through the first task operator to ensure that the data processing task is initialized only when the structure of the first task data is the required structure.

[0131] In some embodiments, the management center may send the verification information to the first task operator. Alternatively, when the data processing task is a distributed task, the management center may broadcast the verification information to the first task operator of each data processing task. In this case, each data processing task includes a first task operator and a second task operator, and each first task operator may receive the verification information. When each first task operator receives the verification information, if the verification information indicates a structural change, each first task operator generates a structural change event to initialize the second task operator in each data processing task.

[0132] For ease of description, the data processing task that sends a structure change request is referred to as the first data processing task, and the data processing task that does not send a structure change request is referred to as the second data processing task. In this embodiment, the management center can broadcast verification information to the first task operator of the first data processing task and the first task operator of the second data processing task, so that the second data processing task can initialize the second task operator of the second data processing task in advance if it does not detect any changes in the structure of the first task data.

[0133] When the data processing task is a distributed task, the system framework diagram of this embodiment can be as follows: Figure 7 shown.

[0134] In some embodiments, the management center may broadcast the verification information when the verification information indicates a structural change, so that if the first task operator in the first data processing task receives the verification information through broadcast, it can directly determine that the verification information is verification information indicating a structural change. If the verification information is not received through broadcast, it can directly determine that the verification information is verification information indicating that the structure has not changed. At this time, the management center can only return the verification information to the first task operator in the first data processing task when the verification information indicates that the structure has not changed.

[0135] In some embodiments, when there are multiple first data processing tasks, that is, when the management center receives structure change requests sent by the first task operators in multiple first data processing tasks, the management center can merge the multiple structure change requests to obtain a merged structure change request, and then forward the merged structure change request to the structure configuration center. The structure configuration center then confirms the structure of the first task data based on the merged structure change request.

[0136] In some embodiments, this embodiment further includes:

[0137] If the detection result indicates that the structure of the first task data has not changed, the first task data is written to a corresponding position of the data source through the first type of data processing task.

[0138] Among them, when the detection structure indicates that the structure has not changed, it means that the structure of the first task data and the structure of the data in the data processing task are the same. Through the first category in the data processing task, the first task data can be written to the corresponding position. Therefore, in this embodiment, if the detection result indicates that the structure has not changed, the first task data is written to the corresponding position through the data processing task.

[0139] When the data processing task includes a first task operator and a second task operator, the first task operator can be used to detect the structure of the first task data. If the detection result indicates that the structure of the first task data has not changed, a data event is generated and sent to the second task operator. The second task operator extracts the first task data from the data event and writes the first task data to the corresponding location.

[0140] As can be seen from the above, in an embodiment of the present application, the first task data to be processed by the data processing task is read from the data source, the structure of the first task data is detected, and a detection result is obtained; if the detection result indicates that the structure of the first task data has changed, the first category in the data processing task is triggered to be initialized, and the first category includes a data writing function; through the first category in the initialized data processing task, the first task data is written to the corresponding position of the data source, so that when the structure of the first task data changes, there is no need to restart the data processing task, and the first task data can also be successfully written to the corresponding position of the data source, thereby reducing the probability of data loss and reducing processing delays.

[0141] The following is based on Figure 8 , the data processing method provided by the present application is further described. In this embodiment, the data processing task includes a first task operator, a second task operator, and a third task operator as an example, and the cache space is used as the corresponding position for explanation.

[0142] 801. Read first task data to be processed from a data source through a first task operator.

[0143] 802. Determine, through the first task operator, whether the structure of the historical task data is the same as the structure of the first task data.

[0144] 803. If the structure of the historical first task data is the same as the structure of the first task data, a detection result indicating that the structure of the first task data has not changed is obtained.

[0145] 804. If the detection result indicates that the structure of the first task data has not changed, the first task data is transferred to the second task operator.

[0146] 805 . Determine the partition identifier corresponding to the first task data in the cache space through the second task operator, and send the first task data and the partition identifier to the third task operator.

[0147] 806. Write the first task data into the partition corresponding to the partition identifier in the cache space through the third task operator.

[0148] The historical task data can be the data last written to the cache before the first task data. For example, if the historical task data is data a0 and the first task data obtained is data a1, the structure of data a1 is the same as that of data a0. When data a1 is written to the cache, the historical task data becomes data a1.

[0149] It can be understood that after the first task data is written into the cache space, the first task data can be referred to as second task data in the cache space.

[0150] 807. If the structure of the historical task data is different from the structure of the first task data, a detection result indicating that the structure of the first task data has changed is obtained.

[0151] 808. If the detection result indicates that the structure of the first task data has changed, a structure change request is sent to the management center through the first task operator.

[0152] 809. The management center forwards the structure change request to the structure configuration center.

[0153] 8010. The structure configuration center verifies the structure of the first task data in the structure change request, obtains verification information, and returns the verification information to the management center.

[0154] 8011. When the verification information indicates a structural change, the management center broadcasts the verification information so that each first task operator can receive the verification information.

[0155] 8012. When the verification information indicates a structure change, generate a structure change event through the first task operator.

[0156] It is understandable that before the verification information is received by the first task operator, the first task operator pauses obtaining new first task data from the data source.

[0157] 8013. Send the first task data and the structure change event to the second task operator through the first task operator.

[0158] Among them, a queue can be created between the first task operator and the second task operator. Through the first task operator, the structure change event is first stored in the queue, and then the first task data is stored in the queue, so that through the second task operator, the structure change event is first obtained from the queue, and then the first task data is obtained from the queue.

[0159] After the first task data is stored in the queue through the first task operator, new first task data continues to be obtained from the data source through the first task operator.

[0160] Optionally, the first task operator may generate a data event (DataEvent) according to the first task data, and then pass the data event to the second task operator.

[0161] 8014. When a structure change event is received through the second task operator, the first subclass in the second task operator is initialized to obtain an initialized second task operator, and the structure change event is sent to the third task operator.

[0162] 8015. When a structure change event is received through the third task operator, the second task data in the cache space is written to the data lake through the second subclass in the third task operator, and the second subclass of the third task operator in the data processing task is initialized to obtain the initialized third task operator.

[0163] It is understandable that the manner of initializing the first subclass in the second task operator and the second subclass in the third task operator may refer to the process of initializing the first class, and this embodiment will not be repeated here.

[0164] 8016. Determine the partition identifier corresponding to the first task data in the cache space through the first subclass of the initialized second task operator, and send the first task data and the partition identifier to the initialized third task operator.

[0165] 8017. Write the first task data into the partition corresponding to the partition identifier in the cache space through the initialized second subclass of the third task operator.

[0166] For example, the first task operator reads the first task data a1 to be processed from the data source. The structure of the first task data a1 includes two fields: id and name. If the structure of the first task data a1 is different from that of the historical task data, a structure change request ("id, name", 1, 1724482543914) is sent to the management center. At the same time, the thread corresponding to the first task operator is blocked and the reading of new first task data from the data source is suspended. The management center sends the structure change request to the structure configuration center. If the structure configuration center finds that the structure of the first task data a1 in the structure change request is the required structure, it generates verification information ("id, name", 1724482543914, 1, true) and returns the verification information to the management center. The management center broadcasts the verification information to the first task operator.

[0167] After receiving the verification information ("id, name", 1724482543914, 1, true) through the first task operator, the first task data a1 and the structure change event are sent to the second task operator through the first task operator, and the new first task data is continued to be obtained through the first task operator.

[0168] When a structure change event is received through the second task operator, the first subclass in the second task operator is initialized according to the structure change event, and the structure change event is sent to the third task operator. According to the structure change event, the second task data in the cache space is written to the data lake through the third task operator, and the second subclass in the third task operator is initialized.

[0169] The partition identifier corresponding to the first task data a1 is determined through the first subclass in the initialized second task operator, and the first task data a1 is written to the partition corresponding to the partition identifier in the cache space through the second subclass in the initialized third task operator.

[0170] If the structure of the first task data a1 is the same as the structure of the historical task data, the first task data a1 is sent to the second task operator through the first task operator, and the partition identifier corresponding to the first task data a1 is determined through the first subclass in the second task operator. The first task data a1 and the partition identifier are sent to the third task operator, and the first task data a1 is written to the partition corresponding to the partition identifier in the cache space through the second subclass of the third task operator.

[0171] The specific implementation method and corresponding beneficial effects of this embodiment can be specifically referred to the above method embodiment, and this embodiment will not be repeated here.

[0172] It should be understood that, although each step in the flowcharts involved in the above-mentioned embodiments is displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0173] Based on the same inventive concept, the present application also provides a data processing device for implementing the above-mentioned data processing method. The implementation solution provided by the device is similar to the implementation solution described in the above-mentioned method. Therefore, the specific limitations of the embodiments of one or more data processing devices provided below can be referred to the above-mentioned limitations of the data processing method, and the specific limitations are not repeated here.

[0174] This embodiment also provides a data processing device, which can be integrated into a terminal device. Figure 9 As shown, the data processing device 900 may include:

[0175] The reading module 901 is configured to read first task data to be processed by a data processing task from a data source.

[0176] The detection module 902 is configured to detect the structure of the first task data and obtain a detection result.

[0177] The trigger module 903 is configured to trigger the initialization of the first type of data processing tasks if the detection result indicates that the structure of the first task data has changed, and the first type includes a data writing function.

[0178] The writing module 904 is configured to write the first task data into a corresponding position of the data source through the first type of the initialized data processing task.

[0179] The above-mentioned data processing device reads the first task data to be processed by the data processing task from the data source, detects the structure of the first task data, and obtains the detection result; if the detection result indicates that the structure of the first task data has changed, the first category in the data processing task is triggered to be initialized, and the first category includes a data writing function; through the first category in the initialized data processing task, the first task data is written to the corresponding position of the data source, so that when the structure of the first task data changes, there is no need to restart the data processing task, and the first task data can also be successfully written to the corresponding position of the data source, thereby reducing the probability of data loss and reducing processing delays.

[0180] In some embodiments, the trigger module 903 is specifically configured to execute:

[0181] Based on the structure of the first task data, updating the data structure configured in the first category of data processing tasks;

[0182] A target instance is created in the updated first class, and the data structure configured in the target instance is the same as the structure of the first task data.

[0183] The write module 904 is specifically used to execute:

[0184] The first task data is written to the corresponding position of the data source through the target instance of the first type of the initialized data processing task.

[0185] In some embodiments, the trigger module 903 is specifically configured to execute:

[0186] Based on the structure of the first task data, a new constructor is created in the first class of data processing tasks to update the data structure configured in the first class of data processing tasks. The data structure configured in the new constructor is the same as the structure of the first task data.

[0187] In some embodiments, the trigger module 903 is specifically configured to execute:

[0188] Determine whether the existing constructors in the first category of data processing tasks include a constructor of the structure of the first task data;

[0189] If it does not exist, a new constructor is created in the first class in the data processing task based on the structure of the first task data.

[0190] In some embodiments, the write module 904 is further configured to execute:

[0191] If there is a constructor including a structure of the first task data among existing constructors in the first class of the data processing task, a target instance is created based on the constructor including the structure of the first task data.

[0192] In some embodiments, the trigger module 903 is specifically configured to execute:

[0193] Based on the structure of the first task data, the constructor in the first class of the data processing task is modified so that the data structure configured in the modified constructor is the same as the structure of the first task data;

[0194] Create a target instance based on the modified constructor in the updated first class.

[0195] In some embodiments, the write module 904 is further configured to execute:

[0196] The second task data stored in the cache space is written to the data lake through the first type target instance in the data processing task.

[0197] In some embodiments, the data processing task includes a first task operator and a second task operator, the first task operator is used to detect the structure of the first task data, and the second task operator includes a first category. The trigger module 903 is specifically used to execute:

[0198] If the detection result indicates that the structure of the first task data has changed, a structure change event is generated by the first task operator, and the structure change event is transmitted to the second task operator;

[0199] When a structure change event is received through the second task operator, the first class is triggered to be initialized.

[0200] In some embodiments, the trigger module 903 is specifically configured to execute:

[0201] If the detection result indicates that the structure of the first task data has changed, a structure change request is sent to the management center through the first task operator;

[0202] Receive verification information returned by the management center based on the structure change request;

[0203] In the case where the verification information indicates a structural change, a structural change event is generated by the first task operator.

[0204] In some embodiments, the write module 904 is further configured to execute:

[0205] If the detection result indicates that the structure of the first task data has not changed, the first task data is written to a corresponding position of the data source through the first type of data processing task.

[0206] In some embodiments, the detection module 902 is configured to perform:

[0207] Get the structure of historical task data;

[0208] If the structure of the historical task data is the same as the structure of the first task data, a detection result indicating that the structure of the first task data has not changed is obtained;

[0209] If the structure of the historical task data is different from the structure of the first task data, a detection result indicating that a change has occurred in the structure of the first task data is obtained.

[0210] Based on the same inventive concept, an embodiment of the present application further provides a computer device, which may be a server or a terminal device. The computer device includes a memory and a processor. The memory stores a computer program, and the processor implements the steps of the above-mentioned data processing method when executing the computer program. This implements various functions, such as:

[0211] Reading first task data to be processed by the data processing task from a data source;

[0212] Detecting the structure of the first task data to obtain a detection result;

[0213] If the detection result indicates that the structure of the first task data has changed, the first type of data processing tasks is triggered to be initialized, and the first type includes a data writing function;

[0214] The first task data is written to the corresponding position of the data source through the first type of the initialized data processing task.

[0215] In an embodiment of the present application, first task data to be processed by a data processing task is read from a data source, and the structure of the first task data is detected to obtain a detection result; if the detection result indicates that the structure of the first task data has changed, the first category in the data processing task is triggered to be initialized, and the first category includes a data writing function; through the first category in the initialized data processing task, the first task data is written to the corresponding position of the data source, so that when the structure of the first task data changes, there is no need to restart the data processing task, and the first task data can also be successfully written to the corresponding position of the data source, thereby reducing the probability of data loss and reducing the occurrence of processing delays.

[0216] The specific implementation of the above operations can be found in the previous embodiments and will not be repeated here.

[0217] In one embodiment, the computer device is a terminal device, for example, its internal structure diagram can be as follows Figure 10As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface, the display unit and the input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a data processing method is implemented. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse, etc.

[0218] Those skilled in the art will understand that Figure 10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0219] Based on the same inventive concept, an embodiment of the present application also provides a computer-readable storage medium, which may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0220] Since the computer program stored in the computer-readable storage medium can execute any data processing method provided in the embodiments of the present application, the beneficial effects that can be achieved by any data processing method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0221] Based on the same inventive concept, embodiments of the present application further provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in various optional implementations of the above embodiments.

[0222] It should be noted that the object data (including but not limited to user device information, user personal information, etc.) and conversation data involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions. Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods.

[0223] Any reference to the memory, database or other media used in the various embodiments provided herein may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0224] The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may include, but are not limited to, general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic units, data processing logic units based on quantum computing, and the like.

[0225] In the above-mentioned data processing apparatus, computer-readable storage medium, computer device, and computer program product embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a particular embodiment, reference can be made to the relevant descriptions of other embodiments. Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes and beneficial effects of the above-described data processing apparatus, computer-readable storage medium, computer program product, computer device, and their corresponding units can be referred to as described in the data processing method in the above embodiments, and the details will not be repeated here.

[0226] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0227] The above is a detailed introduction to a data processing method, a computer-readable storage medium, and a computer program product provided in the embodiments of the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, based on the ideas of the present application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A data processing method, characterized in that: The method comprises: Reading first task data to be processed by the data processing task from a data source; Detecting the structure of the first task data to obtain a detection result; If the detection result indicates that the structure of the first task data has changed, triggering initialization of a first category of the data processing tasks, the first category including a data writing function; The first task data is written into a corresponding position of the data source through the first type of initialized data processing tasks.

2. The method according to claim 1, wherein Triggering the initialization of the first type of data processing tasks includes: Based on the structure of the first task data, updating the data structure configured in the first category of the data processing task; Creating a target instance in the updated first class, wherein the data structure configured in the target instance is the same as the structure of the first task data; Writing the first task data into a corresponding position of the data source through the first type of the initialized data processing task includes: The first task data is written to a corresponding position of the data source through the target instance of the first type of the initialized data processing task.

3. The method according to claim 2, wherein The updating of the data structure configured in the first category of the data processing task based on the structure of the first task data includes: Based on the structure of the first task data, a new constructor is created in the first class of the data processing task to update the data structure configured in the first class of the data processing task. The data structure configured in the new constructor is the same as the structure of the first task data.

4. The method according to claim 3, wherein The step of creating a new constructor in the first class of the data processing task based on the structure of the first task data includes: Determine whether the existing constructors in the first category of the data processing task include the constructor of the structure of the first task data; If it does not exist, a new constructor is created in the first class in the data processing task based on the structure of the first task data.

5. The method according to claim 2, wherein The updating of the data structure configured in the first category of the data processing task based on the structure of the first task data includes: Based on the structure of the first task data, modifying the constructor in the first class of the data processing task; after the modification, the data structure configured in the constructor is the same as the structure of the first task data; The step of creating a target instance in the updated first class includes: Create a target instance based on the modified constructor in the updated first class.

6. The method according to claim 2, wherein The corresponding position is a cache space, and before updating the data structure configured in the first category of the data processing task based on the structure of the first task data, the method further includes: The second task data stored in the cache space is written into the data lake through the target instance of the first type in the data processing task.

7. The method according to claim 1, wherein The data processing task includes a first task operator and a second task operator, the first task operator is used to detect the structure of the first task data, and the second task operator includes the first category; If the detection result indicates that the structure of the first task data has changed, triggering the first type of the data processing task to be initialized includes: If the detection result indicates that the structure of the first task data has changed, generating a structure change event through the first task operator, and transmitting the structure change event to the second task operator; When the structure change event is received through the second task operator, the first class is triggered to be initialized.

8. The method according to claim 7, wherein If the detection result indicates that the structure of the first task data has changed, generating a structure change event through the first task operator includes: If the detection result indicates that the structure of the first task data has changed, sending a structure change request to the management center through the first task operator; receiving verification information returned by the management center based on the structure change request; In a case where the verification information indicates a structural change, a structural change event is generated by the first task operator.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the steps of the data processing method according to any one of claims 1 to 8.

10. A computer program product, characterized in that The computer program product includes a computer program or instructions, and the computer program or instructions are used by a processor to execute the steps of the data processing method according to any one of claims 1 to 8.

Images