Data processing method, device and system and storage medium

Abstract

The invention discloses a data processing method, device and system and a storage medium. The method relates to the field of big data, and comprises the steps that a target business script of a target job and data source information corresponding to the target business script are acquired, and the data source information at least comprises information of a data source on which the target business script depends when the target business script is executed; performing data source registration processing in the stream processing module based on the data source information; the target service script is sent to a stream processing module, and the stream processing module executes the target service script based on the registered data source information under the condition that the target service script is received. Through the method and the device, the problem of low operation execution efficiency of the stream processing module caused by the fact that a complete script needs to be pre-defined in business operation of the stream processing module in related technologies and a new script cannot be dynamically loaded is solved.

Description

Technical Field

[0001] This application relates to the field of big data, and more specifically, to a data processing method, apparatus, system, and storage medium. Background Technology

[0002] In the current field of real-time big data analysis and processing, especially in systems employing stream processing frameworks, the execution of business jobs relies on pre-written, complete scripts. These scripts typically contain all the logic for data processing, from data source definition, data cleaning, aggregation calculations to the final output of results; each step must be fixed before the job starts. While this approach ensures the stability and predictability of job execution, it proves particularly rigid and inflexible when facing rapidly changing business needs or real-time updated data processing rules.

[0003] Specifically, the stream processing modules in these technologies are designed using a static script execution model. Once a job starts running, its processing logic is fixed and cannot be changed until the job ends or the system restarts. This means that if business rules change, such as adjusting real-time monitoring metrics, updating risk control strategies, or altering data processing flows, the existing job must be stopped, the script manually modified, and then the job restarted. This process is not only time-consuming but also interrupts data processing, reducing system responsiveness and real-time performance, resulting in low efficiency for stream processing modules.

[0004] There is currently no effective solution to the aforementioned problems in the relevant technologies. Summary of the Invention

[0005] The main objective of this application is to provide a data processing method, apparatus, system, and storage medium to solve the problem that the business operations of the stream processing module in the related technology require predefined complete scripts and cannot dynamically load new scripts, resulting in low efficiency in the execution of the stream processing module.

[0006] To achieve the above objectives, according to one aspect of this application, a data processing method is provided. The method includes: obtaining a target business script for a target job and corresponding data source information for the target business script, wherein the data source information includes at least information about the data sources on which the target business script depends during execution; performing data source registration processing in a stream processing module based on the data source information; and sending the target business script to the stream processing module, wherein the stream processing module, upon receiving the target business script, executes the target business script based on the registered data source information.

[0007] Optionally, the data processing method further includes: detecting whether a new message has been added to the target topic in the message middleware; if a new message is detected in the target topic, retrieving the new message from the target topic to obtain the target message, wherein the target message includes indicator information and data source information, and the indicator information is the information required to trigger the execution of the target business script; and determining the target business script and the data source information corresponding to the target business script based on the target message.

[0008] Optionally, the data processing method further includes: parsing the target message to obtain indicator information and data source information; determining the business script that matches the indicator information from the preset script matching rules to obtain the target business script, wherein the script matching rules include triggering conditions for multiple business scripts.

[0009] Optionally, the data processing method further includes: detecting whether the target message was successfully acquired to obtain a first detection result; detecting whether the target message was successfully parsed to obtain a second detection result; detecting whether the target business script was successfully executed to obtain a third detection result; and generating an early warning message if there is a detection result indicating failure among the first, second, and third detection results.

[0010] Optionally, the data processing method further includes: after detecting whether the target business script has been successfully executed and obtaining a third detection result, if the third detection result indicates that the target business script has failed to execute, generating a target log and stopping the execution of the target business script; determining whether there is a business script after the target business script; and executing the business script after the target business script if there is a business script after the target business script.

[0011] Optionally, the data processing method further includes: parsing the data source information to obtain the address information and other information of the data source, wherein the other information includes at least one of the following: data type, data format, access control mechanism; registering the address information and other information to the connector of the stream processing module, wherein the connector is used to establish a data transmission channel between the stream processing module and the data source.

[0012] Optionally, the data processing method further includes: determining whether there are multiple target business scripts; if there are multiple target business scripts, determining the execution order of the target business scripts based on their generation time; and sending the target business scripts and their execution order to the stream processing module.

[0013] To achieve the above objectives, according to another aspect of this application, a data processing system is provided. The system includes: a processing module, configured to acquire a target business script for a target job and corresponding data source information for the target business script, wherein the data source information includes at least information about the data source on which the target business script depends during execution; perform data source registration processing in a stream processing module based on the data source information; and send the target business script to the stream processing module; the stream processing module, configured to execute the target business script based on the registered data source information upon receiving the target business script.

[0014] To achieve the above objectives, according to another aspect of this application, a data processing apparatus is provided. The apparatus includes: an acquisition unit for acquiring a target business script for a target job and corresponding data source information for the target business script, wherein the data source information includes at least information about the data source on which the target business script depends during execution; a registration unit for performing data source registration processing in a stream processing module based on the data source information; and a sending unit for sending the target business script to the stream processing module, wherein the stream processing module, upon receiving the target business script, executes the target business script based on the registered data source information.

[0015] Optionally, the acquisition unit further includes: a detection subunit, used to detect whether a new message has been added to the target topic in the message middleware; an acquisition subunit, used to acquire the new message from the target topic when a new message is detected in the target topic, to obtain the target message, wherein the target message includes indicator information and data source information, and the indicator information is the information required to trigger the execution of the target business script; and a first determination subunit, used to determine the target business script and the data source information corresponding to the target business script based on the target message.

[0016] Optionally, the first determining subunit further includes: a parsing module, used to parse and process the target message to obtain indicator information and data source information; and a determining module, used to determine the business script that matches the indicator information from the preset script matching rules to obtain the target business script, wherein the script matching rules include triggering conditions for multiple business scripts.

[0017] Optionally, the data processing device further includes: a first detection unit for detecting whether the target message has been successfully acquired and obtaining a first detection result; a second detection unit for detecting whether the target message has been successfully parsed and obtaining a second detection result; a third detection unit for detecting whether the target business script has been successfully executed and obtaining a third detection result; and a first generation unit for generating early warning information when there are detection results indicating failure among the first, second, and third detection results.

[0018] Optionally, the data processing device further includes: a second generation unit, configured to generate a target log and stop executing the target business script when the third detection result indicates that the target business script has failed to execute; a judgment unit, configured to determine whether a business script exists after the target business script; and an execution unit, configured to execute the business script that exists after the target business script when a business script exists after the target business script.

[0019] Optionally, the registration unit further includes: a parsing subunit, used to parse the data source information to obtain the address information and other information of the data source, wherein the other information includes at least one of the following: data type, data format, access control mechanism; and a registration subunit, used to register the address information and other information to the connector of the stream processing module, wherein the connector is used to establish a data transmission channel between the stream processing module and the data source.

[0020] Optionally, the sending unit further includes: a judgment subunit for judging whether there are multiple target service scripts; a second determination subunit for determining the execution order of the target service scripts based on their generation time when there are multiple target service scripts; and a sending subunit for sending the target service scripts and their execution order to the stream processing module.

[0021] To achieve the above objectives, according to another aspect of this application, a computer-readable storage medium is provided, which includes a stored executable program, wherein, when the executable program is running, it controls the device where the computer-readable storage medium is located to perform the above-described data processing method.

[0022] To achieve the above objectives, according to another aspect of this application, an electronic device is provided, the electronic device including a memory storing an executable program; and a processor for running the program, wherein the program executes the above-described data processing method during runtime.

[0023] To achieve the above objectives, according to another aspect of this application, a computer program product is provided, including computer instructions that, when executed by a processor, implement the steps of the data processing method described above.

[0024] In this embodiment, by obtaining the target business script of the target job and the corresponding data source information, and performing data source registration processing in the stream processing module based on the data source information, dynamic registration based on the data source information is achieved during job runtime. This allows the job to dynamically adapt to changes in the data source without needing to restart due to data source updates. By sending the target business script to the stream processing module, which then executes the target business script based on the registered data source information, new business logic can be executed without restarting the stream processing module, thereby effectively improving the efficiency of the stream processing module in executing jobs.

[0025] Therefore, the method provided in this application achieves the goal of dynamically registering data sources and executing new scripts in the stream processing module, thereby improving the efficiency of the stream processing module in executing jobs. It also solves the technical problem in related technologies where the business jobs of the stream processing module require pre-defined complete scripts and cannot dynamically load new scripts, resulting in low efficiency in the execution of jobs by the stream processing module. Attached Figure Description

[0026] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0027] Figure 1 This is a hardware structure block diagram of a computer terminal provided according to an embodiment of this application;

[0028] Figure 2 This is a flowchart of a data processing method provided according to an embodiment of this application;

[0029] Figure 3 This is a schematic diagram of a data processing method provided according to an embodiment of this application;

[0030] Figure 4 This is a schematic diagram of a data processing apparatus provided according to an embodiment of this application;

[0031] Figure 5 This is a structural block diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0032] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0033] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0034] It should be noted that the information collected in this application (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) are information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of this data all comply with relevant laws, regulations, and standards, necessary confidentiality measures have been taken, and they do not violate public order and good morals. Corresponding access points are provided for users to choose to authorize or refuse. For example, interfaces are set up between this system and relevant users or organizations, providing users with corresponding access points to choose to agree to or refuse automated decision-making results; if the user chooses to refuse, the process proceeds to the expert decision-making stage.

[0035] Example 1

[0036] According to an embodiment of this application, an embodiment of a data processing method is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0037] The method embodiment provided in Embodiment 1 of this application can be executed on a mobile terminal, computer terminal, or similar computing device. Figure 1 A hardware structure block diagram of a computer terminal (or mobile device) for implementing a data processing method is shown. Figure 1As shown, the computer terminal 10 (or mobile device) may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) 102 (processor 102 may include, but is not limited to, a microprocessor (MCU) or a field-programmable gate array (FPGA), etc.), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output (I / O) interface, a Universal Serial Bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0038] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0039] The memory 104 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the data processing method in this embodiment. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby implementing the aforementioned data processing method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0040] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0041] The display can be, for example, a touchscreen liquid crystal display (LCD), which allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0042] Under the aforementioned operating environment, this application provides the following: Figure 2 The data processing method shown. Figure 2 This is a flowchart of the data processing method according to Embodiment 1 of this application.

[0043] Step S201: Obtain the target business script of the target job and the data source information corresponding to the target business script. The data source information includes at least the information of the data source that the target business script depends on when it is executed.

[0044] Optionally, electronic devices, application systems, servers, or other similar devices can be used as the execution subject of this application. In this embodiment, the target processing system can be used as the execution subject to execute the above-described data processing method.

[0045] Depending on the specific application scenario, the specific type of target job can vary. For example, in a real-time network traffic monitoring scenario, a target job can be used to process traffic, etc. In a real-time e-commerce risk control scenario, a target job can be used to analyze risk data, etc. Therefore, in this embodiment, the type of target job is not specifically limited. A target job can be a job currently running in a stream processing module.

[0046] Optionally, the target business script refers to a script that specifically defines the data processing logic and rules, which guides the stream processing job on how to read, process, and output data. The target business script can be an SQL (Structured Query Language) script.

[0047] The data source information includes at least the information about the data sources that the target business script depends on during execution. The data sources that the target business script depends on during execution include at least one of the following: the data source of the input data required by the target business script, and the data source to which the output data of the target business script needs to flow. Data source information may include information such as the type, address, format, and access control mechanism of the data source.

[0048] In an optional embodiment, the system can subscribe to data source information and target business scripts through a message middleware. For example, once there are new business rules or changes to the data source, the relevant configuration center publishes and updates messages through the message middleware, and the system subscribes to the corresponding topics to receive the latest information.

[0049] In another optional embodiment, the system can subscribe to indicator information and data source information through a message middleware. The indicator information is the information required to trigger the execution of the target business script. Therefore, after receiving the indicator information, the system determines the corresponding target business script based on the indicator information.

[0050] Step S202: Perform data source registration processing in the stream processing module based on the data source information.

[0051] Optionally, the stream processing module employs a stream processing framework that supports high-throughput, low-latency real-time data processing. For example, an optional stream processing module could be Apache Flink; alternatively, other types of stream processing modules could also be used, but this embodiment does not impose any specific limitations.

[0052] After acquiring the data source information, the data source is registered in the stream processing module so that the job can read and process data based on the dynamic data source information. For example, upon receiving the data source information, the stream processing module parses out key information such as the data source type, address, and format. Then, based on the data source type, the stream processing module determines the corresponding connector and registers the aforementioned information with the connector in the stream processing module.

[0053] Dynamic registration of data sources allows stream processing jobs to adjust their data inputs and outputs during runtime, adapting to changes in business rules (i.e., target business scripts) without restarting the job to handle new data sources.

[0054] It's important to note that in traditional dataflow frameworks (such as Flink jobs), the data source information is hard-coded into the job configuration during compilation and submission. This means that if the data source changes, the job must be recompiled and resubmitted. However, in this application, dynamic message reception allows for real-time updates of data source binding information, such as database connections, enabling the job to dynamically adapt to changes in the data source at runtime without requiring a restart.

[0055] Step S203: The target business script is sent to the stream processing module. Upon receiving the target business script, the stream processing module executes the target business script based on the registered data source information.

[0056] Optionally, after completing data source registration, the target business script can be sent to the stream processing module. The stream processing module can then execute the script based on the previously dynamically registered data source information. Upon receiving the script, the stream processing module can immediately apply it to the currently running job without interrupting or restarting it, thereby achieving dynamic updates to the job's business logic.

[0057] In this embodiment, by obtaining the target business script of the target job and the corresponding data source information, and performing data source registration processing in the stream processing module based on the data source information, dynamic registration based on the data source information is achieved during job runtime. This allows the job to dynamically adapt to changes in the data source without needing to restart due to data source updates. By sending the target business script to the stream processing module, which then executes the target business script based on the registered data source information, new business logic can be executed without restarting the stream processing module, thereby effectively improving the efficiency of the stream processing module in executing jobs.

[0058] Therefore, the method provided in this application achieves the goal of dynamically registering data sources and executing new scripts in the stream processing module, thereby improving the efficiency of the stream processing module in executing jobs. It also solves the technical problem in related technologies where the business jobs of the stream processing module require pre-defined complete scripts and cannot dynamically load new scripts, resulting in low efficiency in the execution of jobs by the stream processing module.

[0059] Optionally, in the data processing method provided in this application embodiment, obtaining the target business script of the target job and the data source information corresponding to the target business script includes: detecting whether a new message has been added to the target topic in the message middleware; if a new message is detected in the target topic, obtaining the new message from the target topic to obtain the target message, wherein the target message includes indicator information and data source information, and the indicator information is the information required to trigger the execution of the target business script; and determining the target business script and the data source information corresponding to the target business script based on the target message.

[0060] Optionally, message middleware is used for message publishing and subscription, and is a key component for realizing asynchronous communication between systems. In message middleware, a target topic refers to a specific message queue or channel used to transmit information related to the new business logic (i.e., the target business script) added to the target job.

[0061] The system can subscribe to a target topic and listen for new messages within that topic through relevant interfaces. Once a new message is detected in the target topic, the system retrieves the new message from that topic, thus obtaining the target message.

[0062] Optionally, the indicator information is the information required to trigger the execution of the target business script. For example, in the scenario of real-time network traffic monitoring, operators need to dynamically switch traffic analysis rules based on network status. For instance, in an operator's network environment, analysis rules need to be dynamically adjusted based on real-time traffic status, such as identifying abnormal traffic and controlling bandwidth peaks. In this case, the indicator information can be network metrics of the monitored network (e.g., network latency, traffic volume, traffic peaks), thereby determining which business logic to execute (such as identifying abnormal traffic and controlling bandwidth peaks as mentioned above), i.e., determining which business script to execute. By utilizing message middleware to receive SQL scripts (i.e., the target business script) from the configuration center, new analysis logic can be dynamically loaded and executed without restarting the task. Compared to the original method of requiring downtime to update rules, the rule change delay is reduced from 5 minutes to within 10 seconds, significantly improving response efficiency and reducing resource consumption by 45%.

[0063] For example, in the business scenario of real-time risk control in e-commerce, operators need to dynamically update the risk detection SQL logic (such as risk list identification rules). For instance, in e-commerce platforms, risk control strategies need frequent updates to address new types of illegal activities. In this case, the indicator information can be the e-commerce's transaction metrics (such as e-commerce type, maximum daily transaction value, etc.). Based on these transaction metrics, it's determined which business logic to execute (such as different risk control strategies), i.e., which business script to execute. By using a message queue to push new SQL rules (i.e., the target business script) and dynamically injecting them into running jobs, functions such as risk list matching and transaction behavior analysis can be achieved. Compared to the traditional method of deploying each rule independently, the risk control strategy's effective time can be shortened from 2 minutes to 5 seconds. Operations personnel no longer need to frequently submit jobs, reducing overall operations costs by 50%.

[0064] After receiving the target message, the target business script and the corresponding data source information are determined based on the target message.

[0065] It should be noted that by using message middleware to receive information related to the target business script, the timeliness and reliability of information acquisition can be improved, thereby further improving job execution efficiency.

[0066] Optionally, in the data processing method provided in this application embodiment, determining the target business script and the corresponding data source information based on the target message includes: parsing the target message to obtain indicator information and data source information; determining the business script that matches the indicator information from a preset script matching rule to obtain the target business script, wherein the script matching rule includes triggering conditions for multiple business scripts.

[0067] Optionally, upon receiving the target message, the system decodes and parses the data in the message to extract the indicator information used to distinguish and trigger the update, as well as the data source information required for the update job.

[0068] The system has pre-defined script matching rules, which record the trigger conditions corresponding to different business scripts. These rules can be stored in a mapping table, which can be pre-stored in a database, configuration file, or memory. When an indicator matches a trigger condition, the system determines that the indicator matches the corresponding business script, thus identifying that business script as the target business script.

[0069] For example, if the metric information is the network metric mentioned above, then a certain triggering condition could be "network latency is greater than the first value, and traffic volume is greater than the second value", etc.

[0070] It should be noted that by determining the target business script based on preset matching rules and indicator information, the accurate determination of the target business script is achieved.

[0071] Optionally, in the data processing method provided in the embodiments of this application, the method further includes: detecting whether the target message has been successfully acquired to obtain a first detection result; detecting whether the target message has been successfully parsed to obtain a second detection result; detecting whether the target business script has been successfully executed to obtain a third detection result; and generating a warning message if there is a detection result indicating failure among the first detection result, the second detection result, and the third detection result.

[0072] Optionally, the system checks whether the target message was successfully retrieved to obtain a first detection result. For example, when retrieving a new message from a target topic, the return value of the message middleware's consumption function or exception handling logic can be used to determine whether the target message was successfully retrieved. If the consumption function returns normally or does not throw an exception, the retrieval is considered successful. Another example is that after retrieving the target message, the system checks whether the message content is empty and whether the format meets the preset format requirements. If the message content is empty or the format does not meet the preset format requirements, the retrieval is considered to have failed; conversely, if the message content is not empty and the format meets the preset format requirements, the retrieval is considered successful. The first detection result is used to characterize whether the target message was successfully retrieved.

[0073] Optionally, a second detection result is obtained by detecting whether the target message was successfully parsed. For example, during the parsing process of the target message, it is determined whether an error message is generated. If an error message is generated, the parsing is considered to have failed; if no error message is generated, the parsing is considered to have succeeded. Another example is that after parsing the target message and obtaining the indicator information and data source information, it is determined whether there are outliers in the indicator information and data source information. If outliers are found, the parsing is considered to have failed; otherwise, if no outliers are found, the parsing is considered to have succeeded. The second detection result is used to characterize whether the target message was successfully parsed.

[0074] Optionally, a third detection result is obtained by checking whether the target business script has been successfully executed. For example, after the stream processing module executes the target business script, it checks whether the script has executed correctly as expected. This includes checking whether the data processing logic is valid and whether the output is correct (i.e., whether there are outliers). If the data processing logic is valid and the output is correct, the execution is considered successful; otherwise, the execution is considered to have failed. The third detection result is used to characterize whether the target business script has been successfully executed.

[0075] Optionally, an alert message can be sent to the system administrator or monitoring system when any critical step (i.e., the acquisition, parsing, and execution mentioned above) fails. In other words, if a failure is detected at any stage of message acquisition, parsing, or business script execution, the system will generate an alert message to promptly notify relevant personnel or automatically trigger a recovery process.

[0076] In an optional embodiment, a logging and monitoring system can be used to collect and analyze execution results. If a failure is detected, the system can automatically send an alert via email, SMS, or system notification.

[0077] It should be noted that by adopting the above-mentioned early warning mechanism, the system can promptly detect and respond to potential problems, avoid the impact of data processing anomalies on business operations, and improve the reliability of job execution.

[0078] Optionally, in the data processing method provided in the embodiments of this application, after detecting whether the target business script has been successfully executed and obtaining the third detection result, the method further includes: generating a target log and stopping the execution of the target business script if the third detection result indicates that the target business script has failed to execute; determining whether there is a business script after the target business script; and executing the business script after the target business script if there is a business script after the target business script.

[0079] Optionally, when the system detects that the execution of the target business script has failed, it can generate a target log to record the exception information of the script execution and take measures to stop the further execution of the current script to prevent the error from spreading. For example, it can stop the execution of the current script by setting a flag or calling a stop method. The target log can be used to record detailed information about the failure of the target business script execution, such as error type, error message, and timestamp.

[0080] The system checks whether there are other pending business scripts after the currently failed target business script to determine whether to continue executing subsequent scripts. For example, a script execution queue or list could be designed to store all pre-loaded business scripts in the order they were executed. After an execution failure, the system checks the queue to see if there are any subsequent scripts waiting to be executed.

[0081] When the system determines that there are other scripts after the currently failed target business script, it will automatically skip the failed script and continue to execute the next business script in the queue, ensuring that the data processing flow is not completely interrupted.

[0082] It should be noted that by skipping failed scripts and continuing to execute subsequent scripts in the queue, the system can maintain the continuity of the overall data processing flow and the execution efficiency of business logic when encountering local errors, reducing the risk of a complete shutdown caused by a single script exception, thereby improving the reliability of job processing.

[0083] Optionally, in the data processing method provided in this application embodiment, data source registration processing is performed in the stream processing module based on data source information, including: parsing the data source information to obtain the address information and other information of the data source, wherein the other information includes at least one of the following: data type, data format, access control mechanism; registering the address information and other information to the connector of the stream processing module, wherein the connector is used to establish a data transmission channel between the stream processing module and the data source.

[0084] Optionally, the data source information may include various parameters and configuration information required to access a specific data source. Data source information includes address information and other information. Address information refers to the network location information of the data source, such as hostname and port number. Other information includes data type, data format, access control mechanisms, etc., used to guide the data processing and transmission process.

[0085] When the system receives data source information from the target message, it first needs to parse this information to extract the address information of the data source and other key configuration information for data processing, such as data type, format and access control details, in order to prepare for subsequent data source registration.

[0086] After obtaining the address information and other key configuration information for data processing, the address information and other information are registered with the connector of the stream processing module. The connector is a component in the stream processing framework used to establish connections and exchange data with external data sources. Based on the parsed data source information, the system registers this information with the connector in the stream processing module, establishing a data transmission channel between the data source and the stream processing module. This process is the core of dynamic data source registration, designed to allow stream processing jobs to access and process data in real time based on the latest data source information. For example, Flink's Stream Table Environment can be used to dynamically register the registry or data source based on the data source information. The Stream Table Environment is the core interface in Flink used for manipulating table interfaces and FlinkSQL, providing functions such as table registration and SQL execution.

[0087] It should be noted that the above method enables effective dynamic registration of data source information, allowing stream processing jobs to flexibly adapt to changes in data sources. They can access new data sources or update data source access configurations without restarting, thus improving the job's dynamic adaptability and data processing efficiency.

[0088] Optionally, in the data processing method provided in this application embodiment, sending the target business script to the stream processing module includes: determining whether there are multiple target business scripts; if there are multiple target business scripts, determining the execution order of the target business scripts based on the generation time of the target business scripts; and sending the target business scripts and the execution order to the stream processing module.

[0089] Optionally, when there are multiple sets of indicator messages in the target message, or when multiple messages are added to the target topic (also known as the target message topic), the system may match multiple target business scripts.

[0090] Therefore, the system can determine whether there are multiple target business scripts, and if so, determine the execution order of the target business scripts based on their generation time. For example, the earlier a target business script is generated, the earlier it appears in the execution order. Then, the target business scripts and their execution order are sent to the stream processing module, so that the stream processing module can execute the target business scripts sequentially according to this execution order.

[0091] In an optional embodiment, the generation time of the target business script may depend on the generation time of its corresponding indicator information.

[0092] In another alternative embodiment, the generation time within the target business script may depend on the order of its corresponding metric information.

[0093] Optionally, if the target business script is a single file, it can be sent directly to the stream processing module.

[0094] It should be noted that by introducing a dynamic sorting mechanism for target business scripts, the flexibility and reliability of script execution can be effectively improved. This mechanism ensures that even when multiple business scripts need to be updated simultaneously, the system can automatically determine the execution order of the scripts based on their generation time, avoiding data processing anomalies and business logic conflicts caused by improper script order.

[0095] In an alternative embodiment, Figure 3 This is a schematic diagram of a data processing method provided according to an embodiment of this application. Figure 3 An optional application process of this embodiment will be described. For example... Figure 3 As shown, the target processing system receives target messages through a message middleware and checks whether the target message was successfully acquired. If it fails, an early warning message is generated; if it succeeds, the target message is parsed, and the parsing success is checked. If parsing fails, an early warning message is generated; if parsing succeeds, the indicator information and data source information are determined, and the target business script is determined based on the indicator information. Then, data source registration is performed in the stream processing module based on the data source information. The target business script is sent to the stream processing module, which executes the target business script based on the registered data source information. The system checks whether the target business script was successfully executed; if it fails, execution stops; if it succeeds, the script execution result is recorded.

[0096] Therefore, the method provided in this application achieves the goal of dynamically registering data sources and executing new scripts in the stream processing module, thereby improving the efficiency of the stream processing module in executing jobs. It also solves the technical problem in related technologies where the business jobs of the stream processing module require pre-defined complete scripts and cannot dynamically load new scripts, resulting in low efficiency in the execution of jobs by the stream processing module.

[0097] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0098] Example 2

[0099] According to an embodiment of this application, a data processing system for implementing the above-described data processing method is also provided, the system comprising:

[0100] The processing module is used to obtain the target business script of the target job and the data source information corresponding to the target business script. The data source information includes at least the information of the data source that the target business script depends on when it is executed. Based on the data source information, the processing module performs data source registration processing. The processing module then sends the target business script to the processing module.

[0101] The stream processing module is used to execute the target business script based on the registered data source information when the target business script is received.

[0102] Therefore, the method provided in this application achieves the goal of dynamically registering data sources and executing new scripts in the stream processing module, thereby improving the efficiency of the stream processing module in executing jobs. It also solves the technical problem in related technologies where the business jobs of the stream processing module require pre-defined complete scripts and cannot dynamically load new scripts, resulting in low efficiency in the execution of jobs by the stream processing module.

[0103] Example 3

[0104] This application also provides a data processing apparatus. It should be noted that the data processing apparatus of this application can be used to execute the data processing method provided in this application. The data processing apparatus provided in this application will be described below.

[0105] According to an embodiment of this application, an apparatus for implementing the above-described data processing method is also provided, such as... Figure 4 As shown, the device includes:

[0106] The acquisition unit 401 is used to acquire the target business script of the target job and the data source information corresponding to the target business script, wherein the data source information includes at least the information of the data source that the target business script depends on when it is executed.

[0107] Registration unit 402 is used to perform data source registration processing in the stream processing module based on data source information;

[0108] The sending unit 403 is used to send the target business script to the stream processing module, wherein the stream processing module executes the target business script based on the registered data source information upon receiving the target business script.

[0109] In this embodiment, by obtaining the target business script of the target job and the corresponding data source information, and performing data source registration processing in the stream processing module based on the data source information, dynamic registration based on the data source information is achieved during job runtime. This allows the job to dynamically adapt to changes in the data source without needing to restart due to data source updates. By sending the target business script to the stream processing module, which then executes the target business script based on the registered data source information, new business logic can be executed without restarting the stream processing module, thereby effectively improving the efficiency of the stream processing module in executing jobs.

[0110] Therefore, the method provided in this application achieves the goal of dynamically registering data sources and executing new scripts in the stream processing module, thereby improving the efficiency of the stream processing module in executing jobs. It also solves the technical problem in related technologies where the business jobs of the stream processing module require pre-defined complete scripts and cannot dynamically load new scripts, resulting in low efficiency in the execution of jobs by the stream processing module.

[0111] Optionally, in the data processing apparatus provided in this application embodiment, the acquisition unit further includes: a detection subunit, used to detect whether a new message has been added to the target topic in the message middleware; an acquisition subunit, used to acquire the new message from the target topic when a new message is detected in the target topic, to obtain the target message, wherein the target message includes indicator information and data source information, and the indicator information is the information required to trigger the execution of the target business script; and a first determination subunit, used to determine the target business script and the data source information corresponding to the target business script based on the target message.

[0112] Optionally, in the data processing apparatus provided in this application embodiment, the first determining subunit further includes: a parsing module, used to parse and process the target message to obtain indicator information and data source information; and a determining module, used to determine the business script that matches the indicator information from a preset script matching rule to obtain the target business script, wherein the script matching rule includes triggering conditions for multiple business scripts.

[0113] Optionally, in the data processing apparatus provided in the embodiments of this application, the data processing apparatus further includes: a first detection unit, used to detect whether the target message has been successfully acquired and obtain a first detection result; a second detection unit, used to detect whether the target message has been successfully parsed and obtain a second detection result; a third detection unit, used to detect whether the target business script has been successfully executed and obtain a third detection result; and a first generation unit, used to generate warning information when there is a detection result indicating failure among the first detection result, the second detection result, and the third detection result.

[0114] Optionally, in the data processing apparatus provided in the embodiments of this application, the data processing apparatus further includes: a second generation unit, configured to generate a target log and stop executing the target business script when the third detection result indicates that the target business script has failed to execute; a judgment unit, configured to determine whether there is a business script after the target business script; and an execution unit, configured to execute the business script after the target business script when there is a business script after the target business script.

[0115] Optionally, in the data processing apparatus provided in this application embodiment, the registration unit further includes: a parsing subunit, used to parse the data source information to obtain the address information and other information of the data source, wherein the other information includes at least one of the following: data type, data format, access control mechanism; and a registration subunit, used to register the address information and other information to the connector of the stream processing module, wherein the connector is used to establish a data transmission channel between the stream processing module and the data source.

[0116] Optionally, in the data processing apparatus provided in the embodiments of this application, the sending unit further includes: a judging subunit, used to judge whether there are multiple target service scripts; a second determining subunit, used to determine the execution order of the target service scripts based on the generation time of the target service scripts when there are multiple target service scripts; and a sending subunit, used to send the target service scripts and the execution order to the stream processing module.

[0117] It should be noted that the acquisition unit 401, registration unit 402, and sending unit 403 mentioned above correspond to steps S201 to S203 in Embodiment 1. The three modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules or units can be hardware or software components stored in memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n). The above modules can also be part of a device and run in the computer terminal 10 provided in Embodiment 1.

[0118] Example 4

[0119] Embodiments of this application may provide an electronic device. Figure 5 This is a structural block diagram of an electronic device according to an embodiment of this application. Figure 5 As shown, the electronic device may include: one or more ( Figure 5 (Only one is shown) processor 1002, memory 1004, memory controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display.

[0120] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the methods and apparatus in the embodiments of this application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby implementing the above-described methods. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0121] The processor can access information and applications stored in memory via a transmission device to perform the following steps: obtaining the target business script of the target job and the data source information corresponding to the target business script, wherein the data source information includes at least the information of the data source on which the target business script depends during execution; performing data source registration processing in the stream processing module based on the data source information; and sending the target business script to the stream processing module, wherein the stream processing module, upon receiving the target business script, executes the target business script based on the registered data source information.

[0122] The processor can also call the information and application stored in the memory through the transmission device to perform the following steps: detect whether a new message has been added to the target topic in the message middleware; if a new message is detected in the target topic, retrieve the new message from the target topic to obtain the target message, wherein the target message includes indicator information and data source information, and the indicator information is the information required to trigger the execution of the target business script; determine the target business script and the data source information corresponding to the target business script based on the target message.

[0123] The processor can also call the information and application stored in the memory through the transmission device to perform the following steps: parse the target message to obtain indicator information and data source information; determine the business script that matches the indicator information from the preset script matching rules to obtain the target business script, wherein the script matching rules include triggering conditions for multiple business scripts.

[0124] The processor can also call the information and application stored in the memory through the transmission device to perform the following steps: detect whether the target message has been successfully acquired and obtain a first detection result; detect whether the target message has been successfully parsed and obtain a second detection result; detect whether the target business script has been successfully executed and obtain a third detection result; and generate a warning message if there is a detection result indicating failure among the first detection result, the second detection result, and the third detection result.

[0125] The processor can also call the information and application stored in the memory through the transmission device to perform the following steps: after detecting whether the target business script has been successfully executed and obtaining the third detection result, if the third detection result indicates that the target business script has failed to execute, generate a target log and stop executing the target business script; determine whether there is a business script after the target business script; if there is a business script after the target business script, execute the business script after the target business script.

[0126] The processor can also invoke information and applications stored in memory via a transmission device to perform the following steps: The data processing method further includes: parsing the data source information to obtain the address information and other information of the data source, wherein the other information includes at least one of the following: data type, data format, access control mechanism; registering the address information and other information to the connector of the stream processing module, wherein the connector is used to establish a data transmission channel between the stream processing module and the data source.

[0127] The processor can also call the information and application stored in the memory through the transmission device to perform the following steps: The data processing method further includes: determining whether there are multiple target business scripts; if there are multiple target business scripts, determining the execution order of the target business scripts based on the generation time of the target business scripts; and sending the target business scripts and execution order to the stream processing module.

[0128] In this embodiment, by obtaining the target business script of the target job and the corresponding data source information, and performing data source registration processing in the stream processing module based on the data source information, dynamic registration based on the data source information is achieved during job runtime. This allows the job to dynamically adapt to changes in the data source without needing to restart due to data source updates. By sending the target business script to the stream processing module, which then executes the target business script based on the registered data source information, new business logic can be executed without restarting the stream processing module, thereby effectively improving the efficiency of the stream processing module in executing jobs.

[0129] Therefore, the method provided in this application achieves the goal of dynamically registering data sources and executing new scripts in the stream processing module, thereby improving the efficiency of the stream processing module in executing jobs. It also solves the technical problem in related technologies where the business jobs of the stream processing module require pre-defined complete scripts and cannot dynamically load new scripts, resulting in low efficiency in the execution of jobs by the stream processing module.

[0130] Those skilled in the art will understand that Figure 5The structure shown is for illustrative purposes only. Electronic devices can also be smartphones, tablets, handheld computers, mobile internet devices (MIDs), PADs, and other terminal devices. Figure 5 This does not limit the structure of the aforementioned electronic device. For example, electronic devices may also include components that are more... Figure 5 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 5 The different configurations shown.

[0131] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0132] Example 5

[0133] Embodiments of this application also provide a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the data processing method provided in Embodiment 1.

[0134] Optionally, in this embodiment, the storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.

[0135] This application also provides a computer program product, which, when executed on a data processing device, is a program adapted to perform data processing method steps.

[0136] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0137] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0138] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0139] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0140] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0141] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0142] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A data processing method, characterized in that, include: Obtain the target business script of the target job and the data source information corresponding to the target business script, wherein the data source information includes at least the information of the data source that the target business script depends on when it is executed; Based on the data source information, data source registration is performed in the stream processing module; The target business script is sent to the stream processing module, wherein the stream processing module executes the target business script based on the registered data source information upon receiving the target business script.

2. The method according to claim 1, characterized in that, Obtain the target business script for the target job and the data source information corresponding to the target business script, including: Detect whether a new message has been added to the target topic in the message middleware; Upon detecting a new message in the target topic, the new message is retrieved from the target topic to obtain the target message, wherein the target message includes indicator information and the data source information, and the indicator information is the information required to trigger the execution of the target business script; Based on the target message, the target business script and the corresponding data source information are determined.

3. The method according to claim 2, characterized in that, Determining the target business script and the corresponding data source information based on the target message includes: The target message is parsed and processed to obtain the indicator information and the data source information; The target business script is obtained by determining the business script that matches the indicator information from the preset script matching rules, wherein the script matching rules include trigger conditions for multiple business scripts.

4. The method according to claim 3, characterized in that, The method further includes: Detect whether the target message has been successfully acquired, and obtain the first detection result; The second detection result is obtained by detecting whether the target message has been successfully parsed. The third detection result is obtained by checking whether the target business script has been successfully executed. If any of the first, second, or third detection results contain a detection result that fails to represent the target, an early warning message is generated.

5. The method according to claim 4, characterized in that, After detecting whether the target business script has been successfully executed and obtaining the third detection result, the method further includes: If the third detection result indicates that the target business script has failed to execute, a target log is generated, and the execution of the target business script is stopped. Determine whether a business script exists after the target business script; If a business script exists after the target business script, the business script that follows the target business script shall be executed.

6. The method according to claim 1, characterized in that, Based on the data source information, data source registration is performed in the stream processing module, including: Parse the data source information to obtain the address information and other information of the data source, wherein the other information includes at least one of the following: data type, data format, and access control mechanism; The address information and other information are registered to the connector of the stream processing module, wherein the connector is used to establish a data transmission channel between the stream processing module and the data source.

7. The method according to claim 1, characterized in that, Sending the target business script to the stream processing module includes: Determine whether there are multiple target business scripts; When there are multiple target business scripts, the execution order of the target business scripts is determined based on the generation time of the target business scripts; The target business script and the execution order are sent to the stream processing module.

8. A data processing system, characterized in that, include: The processing module is used to obtain the target business script of the target job and the data source information corresponding to the target business script, wherein the data source information includes at least the information of the data source that the target business script depends on when it is executed; perform data source registration processing in the stream processing module based on the data source information; and send the target business script to the stream processing module. The stream processing module is used to execute the target business script based on the registered data source information upon receiving the target business script.

9. A data processing apparatus, characterized in that, include: The acquisition unit is used to acquire the target business script of the target job and the data source information corresponding to the target business script, wherein the data source information includes at least the information of the data source that the target business script depends on when it is executed; The registration unit is used to perform data source registration processing in the stream processing module based on the data source information. A sending unit is used to send the target service script to the stream processing module, wherein, upon receiving the target service script, the stream processing module executes the target service script based on the registered data source information.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored executable program, wherein, when the executable program is executed, it controls the device on which the computer-readable storage medium is located to perform the data processing method according to any one of claims 1 to 7.

Images