Data processing method and device based on computing engine application and electronic equipment

Abstract

The present disclosure provides a data processing method and device based on computing engine application and electronic equipment, relates to the technical field of computers, in particular to the field of cloud computing. The specific implementation scheme is: in response to the current state of the first computing engine Spark application satisfying the preset application creation trigger condition, a second Spark application is established; in response to the second Spark application satisfying the data processing condition, the first Spark application is controlled to stop processing data, and the first subscription point information of the current processing data recorded by the first Spark application is determined; the second Spark application is controlled to start processing data based on the first subscription point information. The present disclosure realizes smooth migration of data between Spark applications by establishing a new Spark application, without the need to re-specify the processing core number and partition number of the old Spark application, thereby avoiding the downstream data fault phenomenon caused by restarting the old Spark application.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, in particular to the field of cloud computing, and more particularly to a data processing method and device based on computing engine application and electronic equipment. BACKGROUND

[0002] Spark streaming is a streaming batch processing engine based on Spark, which is used to realize the streaming processing of real-time data stream. The processing procedure is to process a batch of data at a fixed time interval, and to improve the processing speed by using the cluster processing advantage of Spark, so as to accelerate the data conversion, business processing and data landing.

[0003] When starting a computing engine Spark application (such as a Spark streaming application), the number of processing cores and the number of partitions need to be specified. The number of processing cores and the number of partitions are two important indicators that determine the performance of the Spark application. If a large amount of data occurs at a certain moment in the process of the Spark application processing data, the processing speed of the Spark application will slow down. SUMMARY

[0004] The present disclosure provides a data processing method and device based on computing engine application and electronic equipment.

[0005] According to a first aspect of the present disclosure, a data processing method based on computing engine application is provided, comprising:

[0006] In response to the current state of the first computing engine Spark application satisfying a preset application creation trigger condition, a second Spark application is established;

[0007] In response to the second Spark application satisfying a data processing condition, the first Spark application is controlled to stop processing data, and first subscription point information of the current processing data recorded by the first Spark application is determined;

[0008] The second Spark application is controlled to start processing data based on the first subscription point information.

[0009] According to a second aspect of the present disclosure, a data processing device based on computing engine application is provided, comprising:

[0010] An establishing module, in response to the current state of the first computing engine Spark application satisfying a preset application creation trigger condition, is configured to establish a second Spark application;

[0011] A control module, in response to the second Spark application satisfying a data processing condition, is configured to control the first Spark application to stop processing data;

[0012] determining a first subscription point information of the first Spark application record for processing data currently;

[0013] The control module is further configured to control the second Spark application to start processing data based on the first subscription point information.

[0014] According to a third aspect of the present disclosure, an electronic device is provided, comprising:

[0015] at least one processor; and a memory connected to the at least one processor in communication; wherein

[0016] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the data processing method based on the computing engine application according to the first aspect.

[0017] According to a fourth aspect of the present disclosure, a non-transitory computer readable storage medium storing computer instructions is provided, the computer instructions being used to enable the computer to perform the data processing method based on the computing engine application according to the first aspect.

[0018] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the data processing method based on the computing engine application according to the first aspect.

[0019] According to the technical solution of the present disclosure, when the current state of the first Spark application meets the preset application creation trigger condition, the second Spark application is established. When the second Spark application has data processing conditions, the first Spark application is controlled to stop processing data, and the second Spark application is controlled to start processing data, without the need to re-specify the number of processing cores and the number of partitions of the old Spark application (such as the first Spark application described above), avoiding the downstream data fault phenomenon caused by restarting the old Spark application, and realizing smooth migration of data between Spark applications.

[0020] It should be understood that the contents described in this part are not intended to identify key or important features of the embodiments of the present disclosure, nor to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description.

[0021] BRIEF DESCRIPTION OF DRAWINGS The accompanying drawings are used to better understand the present scheme and do not limit the present disclosure. Among them:

[0022]

[0023] Figure 1is a flow diagram of a data processing method based on a computing engine application according to an embodiment of the present disclosure;

[0024] Figure 2 is a flow diagram of a data processing method based on a computing engine application according to an embodiment of the present disclosure;

[0025] Figure 3 is a flow diagram of a data processing method based on a computing engine application according to an embodiment of the present disclosure;

[0026] Figure 4 is a flow diagram of a data processing method based on a computing engine application according to an embodiment of the present disclosure;

[0027] Figure 5 is a flow diagram of a data processing method based on a computing engine application according to an embodiment of the present disclosure;

[0028] Figure 6 is a structural block diagram of a data processing apparatus based on a computing engine application according to an embodiment of the present disclosure;

[0029] Figure 7 is a structural block diagram of a data processing apparatus based on a computing engine application according to an embodiment of the present disclosure;

[0030] Figure 8 is a structural block diagram of a data processing apparatus based on a computing engine application according to an embodiment of the present disclosure;

[0031] Figure 9 is a structural block diagram of a data processing apparatus based on a computing engine application according to an embodiment of the present disclosure;

[0032] Figure 10 is a structural block diagram of a data processing apparatus based on a computing engine application according to an embodiment of the present disclosure;

[0033] Figure 11 is a block diagram of an electronic device to implement a data processing method based on a computing engine application according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0034] Exemplary embodiments of the present disclosure are described herein with reference to the accompanying drawings, which are provided to help understand various details of the embodiments of the present disclosure, and should not be considered limiting. Accordingly, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Also, descriptions of well-known functions and structures are omitted from the following description for clarity and conciseness.

[0035] In addition, the terms "first", "second", "third" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second", "third" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "multiple" is at least two, for example, two, three, etc., unless otherwise explicitly and specifically limited.

[0036] It should be noted that the Spark application in the present disclosure is a Spark streaming application.

[0037] In the related art, in order to solve the problem of performance degradation of the Spark application caused by the sudden appearance of a large amount of data, the number of processing cores and the number of partitions of the Spark application need to be re-specified, and the Spark application needs to be restarted. Since the Spark application needs a certain restart time to restart, there is no data output during the restart time, resulting in the phenomenon of faulting of downstream data.

[0038] Therefore, the present disclosure provides a data processing method and device based on a computing engine application and an electronic device. Specifically, the data processing method and device based on a computing engine application of the present disclosure are described below with reference to the accompanying drawings.

[0039] Figure 1 is a flow diagram of a data processing method based on a computing engine application according to an embodiment of the present disclosure. It should be noted that the data processing method based on a computing engine application of the present disclosure can be applied to the data processing device based on a computing engine application of the present disclosure, and the data processing device based on a computing engine application can be configured on an electronic device.

[0040] As shown in Figure 1 the data processing method based on a computing engine application can include the following steps:

[0041] Step 101, in response to the current state of the first computing engine Spark application satisfying the preset application creation trigger condition, establishing a second Spark application.

[0042] Optionally, in some embodiments of the present disclosure, the current state of the first Spark application satisfying the preset application creation trigger condition can include that the current processing performance of the first Spark application satisfies the computing power expansion condition, or the computing power reduction condition, or the Spark application satisfies the update condition. If the current state of the first Spark application satisfies the application creation trigger condition, the required parameters for creating the Spark application are determined according to the current state of the first Spark application, and the second Spark application is established according to the parameters. As an example, the required parameters can include, but are not limited to, the number of processing cores and the number of partitions, etc.

[0043] In step 102, in response to the second Spark application satisfying the data processing condition, the first Spark application is controlled to stop processing data, and the first subscription point information of the current processing data recorded by the first Spark application is determined.

[0044] Optionally, the first Spark application can be controlled to stop or start processing data through a subscription switch. In some embodiments of the present disclosure, in response to the second Spark application satisfying the data processing condition, the first Spark application can be controlled to stop processing data by controlling the state of the first subscription switch.

[0045] As an example, when the second Spark application satisfies the data processing condition, that is, the second Spark application has the ability to process data, the state of the first subscription switch is controlled to switch to the off state. The first Spark application stops processing data when it monitors the off state information of the first subscription switch. It should be noted that the first subscription switch is a switch created when the first Spark application is established.

[0046] In addition, when the second Spark application satisfies the data processing condition, the first subscription point information of the current processing data recorded by the first Spark application is also determined. It should be noted that the first subscription point information is the position identifier of the current processing data of the first Spark application. For example, the first Spark application can record the subscription point information of the current processing data every certain period of time when processing data. In this way, when the first Spark application is controlled to stop processing data, the first subscription point information of the current processing data recorded by the first Spark application also needs to be determined, that is, it is necessary to determine which data the first Spark application is currently processing, so that the second Spark application can continue processing from the data.

[0047] In step 103, the second Spark application is controlled to start processing data based on the first subscription point information.

[0048] Optionally, in some embodiments of the present disclosure, a second subscription switch of the second Spark application can be created, through which the second Spark application processes data. Since the first subscription point information indicates the position of the first Spark application processing data, the second Spark application acquires data based on the first subscription point information and continues to process the data that the first Spark application has not processed.

[0049] According to the data processing method based on the computing engine application provided in the embodiments of the present disclosure, when the current state of the first Spark application meets the preset application creation trigger condition, the second Spark application is established. When the second Spark application has the data processing condition, the first Spark application is controlled to stop processing data, and the second Spark application is controlled to start processing data, so that seamless switching of the two Spark applications in processing data can be realized, and the number of processing cores and the number of partitions of the first Spark application do not need to be specified again, thereby avoiding the downstream data fault phenomenon caused by restarting the first Spark application, and realizing smooth migration of data between Spark applications.

[0050] Figure 2 FIG. 1 is a flow diagram of a data processing method based on a computing engine application according to the second embodiment of the present disclosure. As shown in FIG. 1, the method can include the following steps: Figure 2

[0051] In step 201, in response to the current state of the first computing engine Spark application meeting the preset application creation trigger condition, a second Spark application is established.

[0052] Optionally, in some embodiments of the present disclosure, the current state of the first Spark application meeting the preset application creation trigger condition includes the current processing performance of the first Spark application meeting the first computing power expansion condition, or the first computing power reduction condition, or the Spark application meeting the update condition. If the current state of the first Spark application meets the application creation trigger condition, the parameters required for creating the Spark application are determined according to the current state of the first Spark application, and the second Spark application is established according to the parameters.

[0053] ​As an example, when a large amount of data occurs at a certain moment, causing the first Spark application to process slowly and data accumulation occurs, it is considered that the current processing performance of the first Spark application meets the first computing power expansion condition, and a second Spark application with a larger number of processing cores and partitions needs to be established. The first computing power reduction condition and the update condition are the same. If the current data volume is small, using the first Spark application with a larger number of processing cores and partitions to process data will cause waste of resources. In this case, it is considered that the current processing performance of the first Spark application meets the first computing power reduction condition, and a second Spark application with a smaller number of processing cores and partitions needs to be established. If an update request for the first Spark application is received, it can be determined that the first Spark application currently meets the update condition. When the current state of the first Spark application meets the preset application creation trigger condition, the parameters required for creating the Spark application are determined based on the current state of the Spark application, and the second Spark application is established based on the parameters required for creating the Spark application.

[0054] Step 202, creating a second subscription switch of the second Spark application.

[0055] It should be noted that the second subscription switch of the second Spark application can be initialized and created when the second Spark application is established. When the second subscription switch is created, the switch state is in a default closed state. As an example, the format of the second subscription switch can be appId+pipeName=true / false. Wherein, the appId is a unique identifier given to the second Spark application by the Spark cluster, the pipeName is the name of each data pipeline, true means that the subscription switch is in an open state, and false means that the subscription switch is in a closed state. When the subscription switch is in an open state, the Spark application obtains data through the data pipeline identified by the pipeName.

[0056] It should also be noted that the subscription switch of the Spark application can be stored in middleware, such as redis (RemoteDictionary Server), zookeeper (distributed service framework), etc.

[0057] Step 203, in response to the second Spark application meeting the data processing condition, the state of the first subscription switch is switched to a closed state, which is used to indicate that the first Spark application stops processing data. Wherein, the first subscription switch is a switch created when the first Spark application is established.

[0058] As an example, the format of the first subscription switch can be appId+pipeName=true / false. Wherein, the appId is a unique identifier given by the Spark cluster to the first Spark application, the pipeName is the name of each data pipeline, true means that the subscription switch is in an open state, and false means that the subscription switch is in a closed state.

[0059] Optionally, in some embodiments of the present disclosure, the state of the first subscription switch can be controlled to switch to the closed state through a script.

[0060] Step 204, determining the first subscription point information of the current processing data recorded by the first Spark application.

[0061] It should be noted that when processing a batch of data, the Spark application records the subscription point information of the current processing data once every preset time interval, and stores the recorded subscription point information in the middleware. The subscription point information of the current processing data recorded by the first Spark application is determined as the first subscription point information.

[0062] Step 205, controlling the state of the second subscription switch to switch to an open state, which is used to instruct the second Spark application to start processing data based on the first subscription point information.

[0063] Optionally, in some embodiments of the present disclosure, the state of the second subscription switch can be controlled to switch to the open state through a script.

[0064] According to the data processing method based on the computing engine application provided in the embodiments of the present disclosure, the subscription switch corresponding to the Spark application is created when the Spark application is established, and the switching between the Spark applications is realized by controlling the state of the subscription switch. When the current state of the first Spark application meets the preset application creation trigger condition, the second Spark application and the second subscription switch of the second Spark application are established. When the second Spark application has a data processing condition, the state of the first subscription switch of the first Spark application is controlled to switch to the closed state, and the first Spark application is controlled to stop processing data. The state of the second subscription switch of the second Spark application is controlled to switch to the open state, and the second Spark application starts processing data based on the first subscription point information. The present disclosure realizes the flexible switching between the Spark applications by setting the subscription switch of the Spark application, further avoids the downstream data discontinuity phenomenon caused by restarting the first Spark application, and realizes the smooth migration of data between the Spark applications and the lossless expansion, lossless contraction and lossless upgrade of the Spark application.

[0065] It should be noted that when the current state of the first Spark application meets the preset application creation trigger condition, the first Spark application is switched to the second Spark application, and the second Spark application continues to process data based on the first subscription point information. After switching to the second Spark application, the first Spark application can be retained, and when the current processing performance of the second Spark application meets a certain condition, the second Spark application is switched back to the first Spark application. As an example, Figure 3 is a flow diagram of a data processing method based on a computing engine application according to Embodiment Three of the present disclosure. As shown in Figure 3 based on the above embodiment, when the current state of the first Spark application meets the preset application creation trigger condition, the current processing performance of the first Spark application meets the first computing power expansion condition, the data processing method based on the computing engine application can further include the following steps:

[0066] Step 301, retaining the first Spark application.

[0067] Step 302, in response to the current processing performance of the second Spark application meeting the second computing power contraction condition, controlling the second Spark application to stop processing data, and determining the second subscription point information of the current processing data recorded by the second Spark application.

[0068] It should be noted that the second computing power contraction condition can be the same as the first computing power contraction condition, or the second computing power contraction condition can be different from the first computing power contraction condition.

[0069] Step 303, controlling the first Spark application to start processing data based on the second subscription point information.

[0070] According to the data processing method based on the computing engine application of the present disclosure, when the current processing performance of the first Spark application meets the first computing power expansion condition, the second Spark application with larger number of processing cores and partitions is established. After establishing the second Spark application, the first Spark application is retained, and when the current processing performance of the second Spark application meets the second computing power contraction condition, the second Spark application can be switched back to the first Spark application through the first subscription switch and the second subscription switch, without the need to re-establish the second Spark application with smaller number of processing cores and partitions.

[0071] Figure 4 is a flow diagram of a data processing method based on a computing engine application according to Embodiment Four of the present disclosure. As shown in Figure 4As shown, based on the above embodiment, when the current state of the first Spark application meets the preset application creation trigger condition, the data processing method based on the computing engine application can further include the following steps:

[0072] Step 401, retaining the first Spark application.

[0073] Step 402, in response to the current processing performance of the second Spark application meeting the second computing power expansion condition, controlling the second Spark application to stop processing data, and determining third subscription point information of the current processing data recorded by the second Spark application.

[0074] It should be noted that the second computing power expansion condition can be the same as the first computing power expansion condition, or the second computing power expansion condition can be different from the first computing power expansion condition.

[0075] Step 403, controlling the first Spark application to start processing data based on the third subscription point information.

[0076] According to the data processing method based on the computing engine application provided in the embodiments of the present disclosure, when the current processing performance of the first Spark application meets the first computing power contraction condition, a second Spark application with smaller processing core number and partition number is established to avoid resource waste. After the second Spark application is established, the first Spark application is retained, and when the current processing performance of the second Spark application meets the second computing power expansion condition, the second Spark application can be switched back to the first Spark application through the first subscription switch and the second subscription switch, without the need to re-establish a second Spark application with larger processing core number and partition number.

[0077] Figure 5 is a flow diagram of a data processing method based on a computing engine application according to Embodiment Five of the present disclosure. As Figure 5 As shown, based on the above embodiment, the data processing method based on the computing engine application can further include the following steps:

[0078] Step 501, in response to the current state of the first computing engine Spark application meeting the preset application creation trigger condition, establishing a second Spark application.

[0079] Step 502, in response to the second Spark application meeting the data processing condition, controlling the first Spark application to stop processing data, and determining first subscription point information of the current processing data recorded by the first Spark application.

[0080] Step 503, controlling the second Spark application to start processing data based on the first subscription point information.

[0081] At step 504, the first Spark application is deleted.

[0082] It should be noted that step 504 can also be performed after step 502, that is, after the first Spark application is controlled to stop processing data, the first Spark application is deleted.

[0083] According to the data processing method based on the computing engine application provided in the embodiments of the present disclosure, when the current state of the first Spark application meets the preset application creation trigger condition, the second Spark application is established. When the second Spark application meets the data processing condition, the first Spark application is controlled to stop processing data, and the second Spark application is controlled to start processing data, so that seamless switching of data processing of the two Spark applications can be realized, and the processing core number and the partition number of the first Spark application do not need to be specified again, thereby avoiding the downstream data fault phenomenon caused by restarting the first Spark application, and realizing smooth migration of data between Spark applications. After the first Spark application is switched to the second Spark application, the first Spark application can be deleted, thereby saving the storage resources of the Spark application.

[0084] Figure 6 is a structural block diagram of a data processing apparatus based on a computing engine application provided according to Embodiment Six of the present disclosure. As shown in Figure 6 The data processing apparatus based on the computing engine application can include an establishing module 601, a control module 602, and a determining module 603.

[0085] Specifically, the establishing module 601 is configured to, in response to the current state of the first computing engine Spark application meeting a preset application creation trigger condition, establish a second Spark application.

[0086] The control module 602 is configured to, in response to the second Spark application meeting a data processing condition, control the first Spark application to stop processing data.

[0087] The determining module 603 is configured to determine first subscription point information of currently processed data of the first Spark application record.

[0088] The control module 602 is further configured to control the second Spark application to start processing data based on the first subscription point information.

[0089] In some embodiments of the present disclosure, the establishing module 601 is further configured to: determine parameters required for creating the Spark application based on the current state of the first Spark application; and establish the second Spark application based on the parameters required for creating the Spark application.

[0090] In some embodiments of the present disclosure, the control module 602 is further configured to: control the state of the first subscription switch to switch to a closed state, and the closed state is used to instruct the first Spark application to stop processing data. The first subscription switch is a switch created when the first Spark application is established.

[0091] As to the apparatus in the above embodiments, the specific manners in which various modules perform operations have been described in detail in the embodiments of the method, and thus will not be described in detail here.

[0092] According to the data processing apparatus based on computing engine application provided in the embodiments of the present disclosure, when the current state of the first Spark application meets the preset application creation trigger condition, the second Spark application is established. When the second Spark application meets the data processing condition, the first Spark application is controlled to stop processing data, and the second Spark application is controlled to start processing data, so that seamless switching of data processing of two Spark applications can be realized, and the processing core number and the partition number of the first Spark application do not need to be specified again, and the downstream data fault phenomenon caused by restarting the first Spark application is avoided, and smooth migration of data between Spark applications is realized.

[0093] Figure 7 is a structural block diagram of a data processing apparatus based on computing engine application according to Embodiment Seven of the present disclosure. As shown in Figure 7 On the basis of the above embodiments, the data processing apparatus based on computing engine application can further include a creation module 704.

[0094] Specifically, the creation module 704 is configured to create a second subscription switch of the second Spark application.

[0095] The control module 702 is further configured to control the state of the second subscription switch to switch to an open state, and the open state is used to instruct the second Spark application to start processing data based on the first subscription point information.

[0096] In the above embodiments, the specific manners in which various modules perform operations have been described in detail in the embodiments of the method, and thus will not be described in detail here. Figure 7 The modules 701-703 and Figure 6 The modules 601-603 have the same functions and structures.

[0097] As to the apparatus in the above embodiments, the specific manners in which various modules perform operations have been described in detail in the embodiments of the method, and thus will not be described in detail here.

[0098] According to the data processing apparatus based on the computing engine application provided in the embodiments of the present disclosure, a subscription switch corresponding to the Spark application is created when the Spark application is established, and the switching between Spark applications is realized by controlling the state of the subscription switch. When the current state of the first Spark application meets the preset application creation trigger condition, the second Spark application and the second subscription switch of the second Spark application are established. When the second Spark application meets the data processing condition, the state of the first subscription switch of the first Spark application is switched to the closed state, and the first Spark application is controlled to stop processing data. The state of the second subscription switch of the second Spark application is switched to the open state, and the second Spark application starts processing data based on the first subscription point information. The present disclosure realizes the flexible switching between Spark applications by setting the subscription switch of the Spark application, further avoids the downstream data fault phenomenon caused by restarting the first Spark application, and realizes the smooth migration of data between Spark applications and the lossless expansion, lossless contraction and lossless upgrade of the Spark application.

[0099] It should be noted that the current state of the first computing engine Spark application meeting the preset application creation trigger condition can include any one of the following: the current processing performance of the first Spark application meeting the first computing power expansion condition; the current processing performance of the second Spark application meeting the first computing power contraction condition; the first Spark application currently meeting the update condition. Figure 8 FIG. 8 is a structural block diagram of a data processing apparatus based on a computing engine application according to an eighth embodiment of the present disclosure. As shown in Figure 8 When the current state of the first Spark application meets the preset application creation trigger condition, the data processing apparatus based on the computing engine application can further include a first reservation module 805, and the current processing performance of the first Spark application meets the first computing power expansion condition.

[0100] Specifically, the first reservation module 805 is configured to reserve the first Spark application.

[0101] The control module 802 is further configured to, in response to the current processing performance of the second Spark application meeting the second computing power contraction condition, control the second Spark application to stop processing data, and determine the second subscription point information of the current processing data recorded by the second Spark application.

[0102] The control module 802 is further configured to control the first Spark application to start processing data based on the second subscription point information.

[0103] Wherein, Figure 8 The modules 801-804 and Figure 7 The modules 701-704 have the same functions and structures.

[0104] With regard to the apparatus in the above embodiments, the specific manner in which each module performs operations has been described in detail in the embodiments related to the method, and will not be described in detail here.

[0105] According to the data processing apparatus based on the computing engine application provided in the embodiments of the present disclosure, when the current processing performance of the first Spark application meets the first computing power expansion condition, the second Spark application with a larger number of processing cores and partitions is established. After the second Spark application is established, the first Spark application is retained, and when the current processing performance of the second Spark application meets the second computing power contraction condition, the second Spark application can be switched back to the first Spark application through the first subscription switch and the second subscription switch, without the need to re-establish the second Spark application with a smaller number of processing cores and partitions.

[0106] Figure 9 is a structural block diagram of a data processing apparatus based on a computing engine application according to Embodiment Nine of the present disclosure. As Figure 9 indicated, when the current state of the first Spark application meets the preset application creation trigger condition, that is, the current processing performance of the first Spark application meets the first computing power contraction condition, the data processing apparatus based on the computing engine application can further include a second retention module 906.

[0107] Specifically, the second retention module 906 is configured to retain the first Spark application.

[0108] The control module 902 is further configured to, in response to the current processing performance of the second Spark application meeting the second computing power expansion condition, control the second Spark application to stop processing data and determine third subscription point information of the current processing data recorded by the second Spark application.

[0109] The control module 902 is further configured to control the first Spark application to start processing data based on the third subscription point information.

[0110] wherein, Figure 9 The modules 901-905 and Figure 8 801-805 in the apparatus have the same functions and structures.

[0111] With regard to the apparatus in the above embodiments, the specific manner in which each module performs operations has been described in detail in the embodiments related to the method, and will not be described in detail here.

[0112] According to the data processing apparatus based on the computing engine application provided in the embodiment of the present disclosure, when the current processing performance of the first Spark application meets the first computing power shrinkage condition, the second Spark application with smaller processing core number and partition number is established to avoid resource waste. After the second Spark application is established, the first Spark application is retained, and when the current processing performance of the second Spark application meets the second computing power expansion condition, the second Spark application can be switched back to the first Spark application through the first subscription switch and the second subscription switch, and the second Spark application with larger processing core number and partition number does not need to be re-established.

[0113] Figure 10 is a structural block diagram of a data processing apparatus based on a computing engine application according to Embodiment Ten of the present disclosure. As shown in Figure 10 on the basis of the above-mentioned embodiments, the data processing apparatus based on the computing engine application can further comprise a deletion module 1007.

[0114] Specifically, the deletion module 1007 is configured to delete the first Spark application.

[0115] wherein, Figure 10 1001-1006 in Figure 9 901-906 have the same functions and structures.

[0116] As for the apparatus in the above-mentioned embodiments, the specific manner in which each module performs operations has been described in detail in the embodiments related to the method, and will not be described in detail here.

[0117] According to the data processing apparatus based on the computing engine application provided in the embodiment of the present disclosure, when the current state of the first Spark application meets the preset application creation trigger condition, the second Spark application is established. When the second Spark application has the data processing condition, the first Spark application is controlled to stop processing data, and the second Spark application is controlled to start processing data, so that seamless switching of the two Spark applications for processing data can be realized, the processing core number and the partition number of the first Spark application do not need to be re-specified, the downstream data fault phenomenon caused by restarting the first Spark application is avoided, and smooth migration of data between Spark applications is realized. After the first Spark application is switched to the second Spark application, the first Spark application can be deleted, so that the storage resources of the Spark application are saved.

[0118] According to the embodiments of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium and a computer program product.

[0119] As shown in Figure 11 , Figure 11is a block diagram of an electronic device to implement the data processing method based on computing engine application of the embodiments of the present disclosure. The electronic device is intended to represent various forms including digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smart phones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit the implementations of the present disclosure described and / or claimed in this document.

[0120] As shown in Figure 11 The electronic device includes one or more processors 1101, a memory 1102, and an interface that connects the components, including a high-speed interface and a low-speed interface. The components are interconnected using different buses, and can be mounted on a common main board or otherwise installed as needed. The processor can process instructions executed within the electronic device, including instructions stored in the memory or graphics information of the memory to display a GUI on an external input / output device, such as a display device coupled to the interface. In other embodiments, multiple processors and / or buses can be used with multiple memories and multiple storage devices, if necessary. Also, multiple electronic devices can be connected, each providing part of the necessary operations (e.g., as a server array, a group of blade servers, or a multi-processor system). Figure 11 The processor 1101 is taken as an example in the

[0121] The memory 1102 is a non-transitory computer readable storage medium provided by the present disclosure. The memory stores instructions executable by at least one processor, so that the at least one processor executes the data processing method based on the computing engine application provided by the present disclosure. The non-transitory computer readable storage medium of the present disclosure stores computer instructions for causing a computer to execute the data processing method based on the computing engine application provided by the present disclosure.

[0122] The memory 1102 as a non-transitory computer readable storage medium can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as program instructions / modules corresponding to the data processing method based on the computing engine application in the embodiments of the present disclosure (for example, the data processing method based on the computing engine application in the embodiments of the present disclosure). Figure 10The illustrated establishing module 1001, control module 1002, determining module 1003, creating module 1004, first reserving module 1005, second reserving module 1006, and deleting module 1007). The processor 1101 performs various function applications and data processing of the server by running the non-transient software programs, instructions, and modules stored in the memory 1102, i.e., implements the data processing method based on the computing engine application in the above method embodiments.

[0123] The memory 1102 can include a program storage area and a data storage area, where the program storage area can store an operating system, at least one application required by a function, and the data storage area can store data created by the electronic device based on the data processing method based on the computing engine application, etc. In addition, the memory 1102 can include a high-speed random access memory, and can also include a non-transient memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transient solid-state memory device. In some embodiments, the memory 1102 can optionally include a memory disposed remotely with respect to the processor 1101, which can be connected to the electronic device based on the data processing method based on the computing engine application through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0124] The electronic device based on the data processing method based on the computing engine application can also include an input device 1103 and an output device 1104. The processor 1101, the memory 1102, the input device 1103, and the output device 1104 can be connected by a bus or other means, Figure 11 For example, by way of bus connection.

[0125] The input device 1103 can receive input digital or character information, and generate key signal input related to user settings and function control of the electronic device based on the data processing method based on the computing engine application, such as a touch screen, a keypad, a mouse, a trackpad, a touchpad, a pointing stick, one or more mouse buttons, a trackball, a joystick, etc. The output device 1104 can include a display device, an auxiliary lighting device (e.g., an LED), and a tactile feedback device (e.g., a vibration motor), etc. The display device can include but is not limited to a liquid crystal display (LCD), a light-emitting diode (LED) display, and a plasma display. In some embodiments, the display device can be a touch screen.

[0126] Various implementations of the systems and techniques described here can be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0127] These computer programs (also known as programs, software, software applications or code) include machine instructions for the programmable processor, and can be implemented in a high-level procedural and / or object-oriented programming language, and / or in assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus and / or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0128] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0129] The systems and techniques described herein can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described herein, or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0130] The computer system can include clients and servers. The clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server can arise by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also known as cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS service ("Virtual Private Server", or simply "VPS"). The server can also be a server of a distributed system, or a server combined with a blockchain. It should be understood that various forms of flow shown above can be used to reorder, add or delete steps. For example, the steps described in the present application can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved, which are not limited herein.

[0131] The specific embodiments described above do not constitute an limitation on the scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modification, equivalent replacement, and improvement within the spirit and principles of the present disclosure should be included in the scope of the present disclosure.

Claims

1. A data processing method based on a computing engine application, comprising: in response to a current state of a first computing engine Spark application satisfying a preset application creation trigger condition, creating a second Spark application, comprising: determining required parameters for creating the second Spark application according to the current state of the first Spark application, and creating the second Spark application according to the required parameters, the required parameters comprising a number of processing cores and a number of partitions; in response to the second Spark application satisfying a data processing condition, controlling the first Spark application to stop processing data, and determining first subscription point information of currently processed data recorded by the first Spark application; controlling the second Spark application to start processing data based on the first subscription point information.

2. The method of claim 1, wherein, The controlling the first Spark application to stop processing data comprises: controlling a state of a first subscription switch to switch to a closed state, the closed state being used to instruct the first Spark application to stop processing data; wherein the first subscription switch is a switch created when the first Spark application is created.

3. The method of claim 2, further comprising: creating a second subscription switch of the second Spark application; wherein the controlling the second Spark application to start processing data based on the first subscription point information comprises: controlling a state of the second subscription switch to switch to an open state, the open state being used to instruct the second Spark application to start processing data based on the first subscription point information.

4. The method of any one of claims 1 to 3, wherein, The current state of the first computing engine Spark application satisfying the preset application creation trigger condition comprises any one of the following: a current processing performance of the first Spark application satisfying a first computing power expansion condition; a current processing performance of the first Spark application satisfying a first computing power contraction condition; the first Spark application currently satisfying an update condition.

5. The method of claim 4, wherein, The current state of the first Spark application satisfying the preset application creation trigger condition is that the current processing performance of the first Spark application satisfies the first computing power expansion condition; The method further comprises: retaining the first Spark application; in response to a current processing performance of the second Spark application satisfying a second computing power contraction condition, controlling the second Spark application to stop processing data, and determining second subscription point information of currently processed data recorded by the second Spark application; wherein the current processing performance of the second Spark application satisfying the second computing power contraction condition means that a maximum number of processing cores and partitions of the second Spark application currently is greater than that of the first Spark application; controlling the first Spark application to start processing data based on the second subscription point information.

6. The method of claim 4, wherein, The current state of the first Spark application satisfying the preset application creation trigger condition is that the current processing performance of the first Spark application satisfies the first computing power contraction condition; The method further comprises: retaining the first Spark application; in response to the current processing performance of the second Spark application satisfying a second computing power expansion condition, controlling the second Spark application to stop processing data, and determining third subscription point information of current processing data recorded by the second Spark application; wherein the current processing performance of the second Spark application satisfying the second computing power expansion condition means that the current maximum number of processing cores and the number of partitions of the second Spark application are smaller than those of the first Spark application; controlling the first Spark application to start processing data based on the third subscription point information.

7. The method of claim 1, further comprising: deleting the first Spark application.

8. A data processing apparatus based on a computing engine application, comprising: an establishing module, in response to a current state of a first computing engine Spark application satisfying a preset application creation trigger condition, for establishing a second Spark application, including: determining parameters required for creating the Spark application according to the current state of the first Spark application, and establishing the second Spark application according to the required parameters, the required parameters including the number of processing cores and the number of partitions; a control module, in response to the second Spark application satisfying a data processing condition, for controlling the first Spark application to stop processing data; a determination module, for determining first subscription point information of current processing data recorded by the first Spark application; wherein the control module is further configured to control the second Spark application to start processing data based on the first subscription point information.

9. The apparatus of claim 8, wherein, The control module is further configured to: control the state of a first subscription switch to be switched to an off state, the off state being used to indicate that the first Spark application stops processing data; wherein the first subscription switch is a switch created when the first Spark application is established.

10. The apparatus of claim 9, further comprising: a creation module, for creating a second subscription switch of the second Spark application; wherein the control module is further configured to: control the state of the second subscription switch to be switched to an on state, the on state being used to indicate that the second Spark application starts processing data based on the first subscription point information.

11. The apparatus of any of claims 8-10, wherein, The current state of the first computing engine Spark application satisfying the preset application creation trigger condition includes any one of the following: the current processing performance of the first Spark application satisfying a first computing power expansion condition; the current processing performance of the first Spark application satisfying a first computing power contraction condition; the first Spark application currently satisfying an update condition.

12. The apparatus of claim 11, wherein, The current state of the first Spark application satisfying the preset application creation trigger condition is that the current processing performance of the first Spark application satisfies the first computing power expansion condition; the apparatus further comprises: a first reservation module, for reserving the first Spark application; The control module is further configured to, in response to the current processing performance of the second Spark application satisfying a second computing power shrinkage condition, control the second Spark application to stop processing data, and determine second subscription point information of current processing data recorded by the second Spark application; the current processing performance of the second Spark application satisfying the second computing power shrinkage condition means that the maximum number of processing cores and the number of partitions of the second Spark application are greater than those of the first Spark application; The control module is further configured to control the first Spark application to start processing data based on the second subscription point information.

13. The apparatus of claim 11, wherein, The current state of the first Spark application satisfying the preset application creation trigger condition is that the current processing performance of the first Spark application satisfies a first computing power shrinkage condition; the device further includes: A second reservation module configured to reserve the first Spark application. The control module is further configured to, in response to the current processing performance of the second Spark application satisfying a second computing power expansion condition, control the second Spark application to stop processing data, and determine third subscription point information of current processing data recorded by the second Spark application; the current processing performance of the second Spark application satisfying the second computing power expansion condition means that the maximum number of processing cores and the number of partitions of the second Spark application are less than those of the first Spark application. The control module is further configured to control the first Spark application to start processing data based on the third subscription point information.

14. The device of claim 8, further comprising: A deletion module configured to delete the first Spark application.

15. An electronic device, comprising: comprising: at least one processor; and a memory connected with the at least one processor in communication; wherein The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.

16. A non-transitory computer-readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to enable the computer to perform the method of any one of claims 1-7.

17. A computer program product comprising a computer program which, when executed by a processor, implements the method of any one of claims 1-7.

Images