Real-time analysis task generation method, device and equipment and computer storage medium

Abstract

The invention discloses a real-time analysis task generation method, device and equipment and a computer storage medium, and relates to the technical field of Internet of Things. The method comprises the following steps: acquiring a to-be-analyzed data stream, an outer-layer architecture, an inner-layer architecture, an aggregation condition parameter and a dynamic output field; determining an outer-layer architecture of the to-be-analyzed data stream according to a relationship between the message type and the outer-layer architecture, and analyzing a product field and a dynamic field set in the to-be-analyzed data stream based on the outer-layer architecture; determining an inner-layer architecture according to a relationship between the product field and the inner-layer architecture, and analyzing dynamic output field data in the dynamic field set based on the inner-layer architecture, the dynamic field set and the dynamic output field; and generating a real-time analysis task based on the inner-layer architecture, the outer-layer architecture, the product field, the dynamic output field data, the aggregation condition parameters and the SQL statement generation rule. According to the embodiment of the invention, the analysis task is generated based on the two-layer architecture, adjustment can be performed according to the change of the configuration information, and the flexibility and expansibility of task generation are improved.

Description

Technical Field

[0001] The present application belongs to the technical field of Internet of Things, and in particular, relates to a method, device, equipment and computer storage medium for generating real-time analysis tasks. Background Art

[0002] When IoT platforms analyze device data, they usually use offline batch processing due to the large amount of device data and the complexity of device information. However, as more and more scenarios require real-time analysis of device data by IoT platforms, real-time analysis of device data is becoming more and more important.

[0003] When generating existing real-time analysis tasks, users are required to write real-time analysis code based on specific data models and messages for each device according to their own needs. If new devices are added or the data models and messages of existing devices change, people are required to manually rewrite or modify the code, which results in poor flexibility and scalability in generating real-time analysis tasks. Summary of the invention

[0004] The embodiments of the present application provide a method, apparatus, device and computer storage medium for generating a real-time analysis task to solve the problem of poor flexibility and scalability of the real-time analysis task generation method in the prior art.

[0005] In a first aspect, an embodiment of the present application provides a method for generating a real-time analysis task, the method comprising:

[0006] Obtain the data flow and configuration information to be analyzed. The configuration information includes input information, processing information and output information. The input information includes the outer layer architecture and the inner layer architecture. The processing information includes aggregation condition parameters. The output information includes dynamic output fields.

[0007] Determine the outer architecture of the data stream to be analyzed according to the relationship between the message type of the data stream and the outer architecture, and parse the product fields and dynamic field sets in the data stream to be analyzed based on the outer architecture;

[0008] Determine the inner layer architecture according to the relationship between the product field and the inner layer architecture, and parse the dynamic output field data in the dynamic field set based on the inner layer architecture, the dynamic field set and the dynamic output field;

[0009] Determine the SQL statement based on the inner layer architecture, outer layer architecture, product fields, dynamic output field data, aggregation condition parameters and preset structured query SQL statement generation rules;

[0010] Send SQL statements to the computing engine for execution, generating real-time analysis tasks.

[0011] In a second aspect, an embodiment of the present application provides a device for generating a real-time analysis task, the device comprising:

[0012] An acquisition module is used to acquire the data flow to be analyzed and configuration information, the configuration information includes input information, processing information and output information, the input information includes the outer layer architecture and the inner layer architecture, the processing information includes the aggregation condition parameters, and the output information includes the dynamic output field;

[0013] A parsing module, used to determine the outer layer architecture of the data stream to be analyzed according to the relationship between the message type of the data stream and the outer layer architecture, and parse out the product fields and dynamic field sets in the data stream to be analyzed based on the outer layer architecture;

[0014] The parsing module is further used to determine the inner layer architecture according to the relationship between the product field and the inner layer architecture, and parse the dynamic output field data in the dynamic field set based on the inner layer architecture, the dynamic field set and the dynamic output field;

[0015] A determination module, used to determine the SQL statement based on the inner layer architecture, the outer layer architecture, the product field, the dynamic output field data, the aggregation condition parameters and the preset structured query SQL statement generation rules;

[0016] The generation module is used to send SQL statements to the computing engine for execution and generate real-time analysis tasks.

[0017] In a third aspect, an embodiment of the present application provides a terminal device, comprising: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, the method for generating real-time analysis tasks as in the first aspect is implemented.

[0018] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are executed by a processor, the method for generating a real-time analysis task as in the first aspect is implemented.

[0019] In a fifth aspect, an embodiment of the present application provides a computer program product. When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the method for generating a real-time analysis task as in the first aspect.

[0020] The embodiment of the present application provides a method, device, equipment and computer storage medium for generating a real-time analysis task. The method first obtains a data stream to be analyzed and configuration information, wherein the configuration information includes input information, processing information and output information, wherein the input information includes an outer layer architecture and an inner layer architecture, the processing information includes aggregation condition parameters, and the output information includes dynamic output fields. The outer layer architecture is determined according to the relationship between the message type of the data stream to be analyzed and the outer layer architecture type, and the product field and dynamic field set in the data stream to be analyzed are parsed based on the outer layer architecture; the system is enabled to flexibly process data streams of different formats, thereby enhancing the versatility and extensibility of the system. The inner layer architecture is determined according to the relationship between the product field and the inner layer architecture, and the dynamic output field data in the dynamic field set is parsed based on the inner layer architecture, the dynamic field set and the dynamic output field; by parsing the product field and the dynamic field set, dynamically changing attributes or event data can be processed, allowing the system to adapt to changing data features. A structured query SQL statement is determined based on the inner layer architecture, the outer layer architecture, the product field, the dynamic output field data and the aggregation condition parameters. The SQL statement is sent to the computing engine for execution to generate a real-time analysis task. A two-layer architecture is used in combination. The outer layer is used to process fixed structure data fields, and the inner layer is used to process dynamically changing attributes or event data. SQL statements are determined based on the two-layer architecture and can be adjusted instantly according to changes in real-time data streams and configuration information, thereby realizing the dynamic generation of real-time analysis tasks and improving the flexibility and scalability of real-time analysis task generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solution of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0022] Figure 1 It is a flowchart of a method for generating a real-time analysis task provided in an embodiment of the present application;

[0023] Figure 2 It is a flowchart of a method for obtaining configuration information provided in an embodiment of the present application;

[0024] Figure 3 It is a flowchart of a method for determining an SQL statement provided in an embodiment of the present application;

[0025] Figure 4 It is a structural diagram of a device for generating real-time analysis tasks provided in an embodiment of the present application;

[0026] Figure 5 It is a schematic diagram of the structure of the terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by illustrating the examples of the present application.

[0028] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0029] When the IoT platform of the existing technology performs real-time analysis on device data, static analysis methods are usually used. First, developers need to write the code of the real-time task according to their own needs and submit the task to the real-time computing engine for calculation. Once the code is written and deployed, it is difficult to dynamically adjust it according to the changes in the device data model or message. When a new device is added or the data model changes, the user needs to write analysis tasks or modify the code for the new device one by one, which increases the labor cost and the complexity of system updates. Especially when there are many types of devices and complex data models, the maintenance cost increases significantly, and there is a lack of adaptive ability to the addition of new devices or changes in data models. The flexibility and scalability of the real-time analysis task generation method are poor.

[0030] In order to solve the existing technology, the embodiment of the present application provides a method, device, equipment and computer storage medium for generating a real-time analysis task. The method first obtains the data stream to be analyzed and the configuration information, the configuration information includes input information, processing information and output information, the input information includes the outer layer architecture and the inner layer architecture, the processing information includes the aggregation condition parameter, and the output information includes the dynamic output field. The outer layer architecture is determined according to the relationship between the message type of the data stream to be analyzed and the outer layer architecture type, and the product field and the dynamic field set in the data stream to be analyzed are parsed based on the outer layer architecture; the system can flexibly process data streams of different formats, and enhance the versatility and extensibility of the system. The inner layer architecture is determined according to the relationship between the product field and the inner layer architecture, and the dynamic output field data in the dynamic field set is parsed based on the inner layer architecture, the dynamic field set and the dynamic output field; by parsing the product field and the dynamic field set, dynamically changing attributes or event data can be processed, allowing the system to adapt to the changing data characteristics. The structured query SQL statement is determined based on the inner layer architecture, the outer layer architecture, the product field, the dynamic output field data and the aggregation condition parameter. The SQL statement is sent to the computing engine for execution to generate a real-time analysis task. A two-layer architecture is used in combination. The outer layer is used to process fixed structure data fields, and the inner layer is used to process dynamically changing attributes or event data. SQL statements are determined based on the two-layer architecture and can be adjusted instantly according to changes in real-time data streams and configuration information, thereby realizing the dynamic generation of real-time analysis tasks and improving the flexibility and scalability of real-time analysis task generation.

[0031] The following is an introduction to the method for generating real-time analysis tasks provided in an embodiment of the present application in conjunction with the accompanying drawings.

[0032] Figure 1 A flow chart of a method for generating a real-time analysis task provided by an embodiment of the present application is shown. Figure 1 As shown, the method may include the following steps: S101 to S105.

[0033] S101, obtaining the data flow to be analyzed and configuration information, the configuration information includes input information, processing information and output information, the input information includes an outer layer architecture and an inner layer architecture, the processing information includes aggregation condition parameters, and the output information includes dynamic output fields.

[0034] In some embodiments, the input information, processing information and output information of the configuration can be dynamically defined and managed through a configuration file, a database or a management interface. When different users configure the configuration information, they can only select the storage address corresponding to the database to which the user has authority.

[0035] In some embodiments, the input information may further include project, product, equipment range, message type and attribute field, wherein the message type may be divided into two types: attribute and event. The processing information may further include condition settings, group settings and window calculation settings. Among them, the condition settings may support the selection of processing conditions according to the attribute data type, for example, the integer data supports greater than, greater than or equal to, less than, less than or equal to, and not equal to. The output information may further include storage address and storage time limit.

[0036] Users can set configuration information according to different data flows and business requirements. Through the input information, processing information and output information in the configuration information, they can flexibly control the data processing process and customize the configuration according to different requirements. When the data source or analysis requirements change, only the configuration information needs to be updated without modifying the underlying code, thereby improving the scalability and maintainability of the system.

[0037] S102, determining the outer layer architecture of the data stream to be analyzed according to the relationship between the message type of the data stream and the outer layer architecture, and parsing the product field and dynamic field set in the data stream to be analyzed based on the outer layer architecture.

[0038] In some embodiments, the outer layer architecture is used to define fixed structural data fields in the data stream, wherein the outer layer architecture may include fixed fields such as project fields, product fields, and message types.

[0039] The outer layer architecture processes fixed structure data fields, which can convert the original data stream into structured data, reduce unnecessary data in the processing steps, and facilitate subsequent analysis and processing.

[0040] S103, determining the inner layer architecture according to the relationship between the product field and the inner layer architecture, and parsing the dynamic output field data in the dynamic field set based on the inner layer architecture, the dynamic field set and the dynamic output field.

[0041] In some embodiments, the inner layer architecture is used to define dynamically changing attributes or event data in the data stream.

[0042] The inner layer architecture handles dynamically changing attribute or event data, allowing the system to adapt to different data structures and types.

[0043] S104, determining an SQL statement based on the inner layer architecture, the outer layer architecture, the product fields, the dynamic output field data, the aggregation condition parameters and the preset structured query SQL statement generation rules.

[0044] By using a two-tier architecture to determine SQL statements, the system can adapt to changes in data structure and process new or changed data fields without human intervention, allowing the system to respond quickly to data changes and improve the automation of real-time data analysis.

[0045] S105, sending the SQL statement to the computing engine for execution, generating a real-time analysis task.

[0046] In some embodiments, the dynamic allocation of real-time analysis tasks is achieved through the Flink Native Kubernetes platform. When there are multiple tasks and each task requires multiple task managers, TaskManager resources can be dynamically provided to solve the resource scaling problem.

[0047] First, the data stream to be analyzed and the configuration information are obtained. The configuration information includes input information, processing information and output information. The input information includes the outer layer architecture and the inner layer architecture. The processing information includes the aggregation condition parameters. The output information includes the dynamic output field. The outer layer architecture is determined according to the relationship between the message type of the data stream to be analyzed and the outer layer architecture type, and the product field and dynamic field set in the data stream to be analyzed are parsed based on the outer layer architecture; the system can flexibly process data streams of different formats, and enhance the versatility and scalability of the system. The inner layer architecture is determined according to the relationship between the product field and the inner layer architecture, and the dynamic output field data in the dynamic field set is parsed based on the inner layer architecture, the dynamic field set and the dynamic output field; by parsing the product field and the dynamic field set, dynamically changing attributes or event data can be processed, allowing the system to adapt to changing data characteristics. The structured query SQL statement is determined based on the inner layer architecture, the outer layer architecture, the product field, the dynamic output field data and the aggregation condition parameters. The SQL statement is sent to the computing engine for execution to generate a real-time analysis task. A two-layer architecture is used in combination. The outer layer is used to process fixed structure data fields, and the inner layer is used to process dynamically changing attributes or event data. SQL statements are determined based on the two-layer architecture and can be adjusted instantly according to changes in real-time data streams and configuration information, thereby realizing the dynamic generation of real-time analysis tasks and improving the flexibility and scalability of real-time analysis task generation.

[0048] In some embodiments, the data reporting message format of the data stream to be analyzed is shown in Table 1:

[0049] Table 1 Data reporting message format

[0050]

[0051] Data is the specific content of the device message, including the relevant attributes or event information of the outer architecture. The format of the outer architecture is shown in Table 2:

[0052] Table 2 External architecture structure

[0053]

[0054]

[0055] For the outer layer architecture, the outer layer data field key value (key) is a fixed dynamic field set (params), and the corresponding value is the attribute parameter of the product, which will change with the product. The dynamic parsing data precondition is formed by superimposing the outer layer schema and the product model schema.

[0056] In some embodiments, obtaining the data stream to be analyzed may include:

[0057] Get the data source address of the data stream to be analyzed;

[0058] Select the target connector based on the relationship between the data source address and the preset connector;

[0059] The data stream to be analyzed is obtained from the data source corresponding to the data source address of the data stream to be analyzed based on the target connector.

[0060] By selecting the target connector based on the relationship between the data source address and the preset connector, the system can adapt to multiple data sources to meet data acquisition requests of different data source types.

[0061] In some embodiments, the data source may include real-time message middleware such as RabbitMQ or Kafka. Use RabbitMQ Connector or Kafka Connector to connect to real-time message middleware such as RabbitMQ or Kafka respectively and obtain the data stream to be analyzed.

[0062] In some embodiments, Figure 2 As shown, obtaining configuration information may include: S201 to S202.

[0063] S201, obtaining input information, processing information, and output information identifiers selected by a user in a visual interface.

[0064] S202: Acquire corresponding input information, processing information and output information according to the input information, processing information and output information identifiers.

[0065] The complex process of real-time analysis tasks is dynamically generated into real-time tasks through graphical orchestration. This method does not require code development, and users can quickly complete the configuration through simple operations such as dragging and dropping, clicking, etc. Compared with traditional command line or code configuration, the configuration speed is greatly accelerated.

[0066] In some embodiments, when the input information of different tasks is the same but the required processing is different, multiple nodes containing processing information can be configured on the node corresponding to the input information, and each node corresponding to the processing information needs to be configured with a corresponding output information node; when the input information and processing are the same but the output information is different, multiple output information nodes can be configured after the node corresponding to the processing information. The processing node must be connected to the output node, and the input node must be connected to the processing node or the output node.

[0067] In some embodiments, Figure 3 As shown, the configuration information also includes the window calculation mode and the window time. Based on the inner layer architecture, the outer layer architecture, the product field, the dynamic output field data, the aggregation condition parameters and the preset structured query SQL statement generation rules, the SQL statement is determined, which may include the following steps: S301 to S304.

[0068] S301, constructing an SQL statement framework according to the outer layer architecture, dynamic field set, product field and SQL statement generation rules.

[0069] In some embodiments, when the message type is attribute reporting, when reporting dynamic fields to build the SQL statement framework, a multi-level nested structure is required to parse out valid dynamic fields (params), project fields, product fields, etc., but the inner fields cannot be parsed, so they need to be parsed according to the subsequent inner schema.

[0070] S302, generating inner-layer SQL statement fragments according to the inner-layer architecture, dynamic output field data and SQL statement generation rules, and splicing them with the SQL statement framework to obtain SQL query statements.

[0071] In some embodiments, in one example, when the dynamic output field is temperature, the concatenated SQL statement may be cast(data['params']['temp']['value']as bigint)as temp.

[0072] S303, generating a time processing SQL statement fragment according to the window calculation mode, the window time and the SQL statement generation rule, and splicing it with the SQL query statement to obtain an SQL query statement after splicing the time processing SQL statement fragment.

[0073] In some embodiments, the window computing mode may include rolling window computing or sliding window computing. Rolling window computing is to divide the data stream into fixed-size, non-overlapping windows; sliding window computing is to divide the data stream into fixed-size windows with overlapping between adjacent windows.

[0074] In some embodiments, the window time may include event time or processing time. Event time is the time when the event actually occurs, usually represented by a timestamp carried by the data itself; processing time is the time when the data is processed.

[0075] In an example, when the window time is the processing time, the generated time processing SQL statement fragment may be DATE_FORMAT(LOCALTIMESTAMP,'yyyy-MM-ddHH:mm:ss')as insert_time.

[0076] S304, generating a processing SQL statement according to the aggregation condition parameters and the SQL statement generation rule, and splicing it with the SQL query statement after splicing the time processing SQL statement fragments to obtain an SQL statement.

[0077] In some embodiments, the aggregation condition parameters include the dynamic field to be aggregated, the aggregation method, and the name of the aggregation result.

[0078] In some embodiments, aggregation methods may include sum, min, max, average, and count.

[0079] In an example, the dynamic field to be aggregated is temperature, the aggregation method is to find the maximum value, the aggregation result is temperature, and the generated processing SQL statement may be max(cast(data['params']['temp']['value']as bigint))as temp.

[0080] Decompose SQL statements into independent fragments, and dynamically generate different fragments based on runtime data and configuration, which can quickly adapt to changes in data structure and business needs. Automatically generate SQL statements, reduce the workload of manually writing and debugging SQL statements, and improve the flexibility of real-time analysis task generation.

[0081] In some embodiments, the configuration information further includes a project field. After sending the SQL statement to the computing engine and generating the real-time analysis task, the method may further include:

[0082] Concatenate the project field, product field and window time to generate the storage table name;

[0083] Create a database table based on the preset database type and database address corresponding to the storage table name and project field to store the execution results of the real-time analysis task.

[0084] By dynamically concatenating fields to generate storage table names, it can flexibly adapt to the real-time analysis needs of different projects, products and time windows. By presetting database types and addresses, it helps to achieve data isolation management, reduce the risk of data confusion and improve the efficiency of data query.

[0085] In some embodiments, after concatenating the project field, product field and window time to generate a storage table name, the method may further include: converting the storage table name to generate a short table name through a hash algorithm, and setting the short table name as a new storage table name.

[0086] Converting storage table names to short table names can take up less storage space and reduce database parsing and processing time. By using a hash algorithm to generate segment table names, the uniqueness of the table name is ensured.

[0087] In some embodiments, the database table periodically cleans up data according to a preset time to improve the reuse rate of storage space.

[0088] In some embodiments, the data stream to be analyzed is grouped and stored according to preset rules, and SQL statements are sent to the computing engine for execution to generate real-time analysis tasks, which may include:

[0089] According to the relationship between the storage group and the computing engine of the data stream to be analyzed, the SQL statement is sent to the computing engine corresponding to the storage group for execution, generating a real-time analysis task.

[0090] In some embodiments, the Flink TaskManager monitors the message middleware in real time and stores the data stream to be analyzed obtained from the message middleware into a preset storage group.

[0091] By grouping the data streams to be analyzed, we can ensure that only one computing engine uses the same data, avoiding duplicate data processing.

[0092] In some embodiments, Flink Checkpoints are used to periodically save the task status of real-time analysis tasks. When a task fails, the task can be automatically restarted and restored through the most recent checkpoint, thereby saving the task's computing status and performing fault-tolerant processing, ensuring that data correction can be achieved after anomalies occur.

[0093] Figure 4 A device 400 for generating a real-time analysis task provided by an embodiment of the present application is shown. Figure 4 As shown, the device may include:

[0094] The acquisition module 401 is used to acquire the data flow to be analyzed and the configuration information, the configuration information includes input information, processing information and output information, the input information includes the outer layer structure and the inner layer structure, the processing information includes the aggregation condition parameters, and the output information includes the dynamic output field;

[0095] The parsing module 402 is used to determine the outer layer architecture of the data stream to be analyzed according to the relationship between the message type of the data stream and the outer layer architecture, and parse the product field and dynamic field set in the data stream to be analyzed based on the outer layer architecture;

[0096] Correspondingly, the parsing module 402 is further used to determine the inner layer architecture according to the relationship between the product field and the inner layer architecture, and parse the dynamic output field data in the dynamic field set based on the inner layer architecture, the dynamic field set and the dynamic output field;

[0097] The determination module 403 is used to determine the SQL statement based on the inner layer structure, the outer layer structure, the product field, the dynamic output field data, the aggregation condition parameters and the preset structured query SQL statement generation rule;

[0098] The generation module 404 is used to send the SQL statement to the computing engine for execution and generate a real-time analysis task.

[0099] In some embodiments, the acquisition module 401 is further used to obtain the data source address of the data stream to be analyzed;

[0100] The determination module 403 is further used to select a target connector according to the relationship between the data source address and the preset connector;

[0101] The acquisition module 401 is further configured to acquire the data flow to be analyzed from a data source corresponding to the data source address of the data flow to be analyzed based on the target connector.

[0102] In some embodiments, the acquisition module 401 is further used to acquire input information, processing information, and output information identifiers selected by the user in the visual interface;

[0103] The acquisition module 401 is further used to acquire corresponding input information, processing information and output information according to the input information, processing information and output information identifiers.

[0104] In some embodiments, the configuration information also includes a window calculation mode and a window time. The device 400 for real-time analysis task generation may also include:

[0105] The generation module 404 is further used to construct an SQL statement framework according to the outer layer architecture, the dynamic field set, the product field and the SQL statement generation rule;

[0106] A splicing module is used to generate inner-layer SQL statement fragments according to the inner-layer architecture, dynamic output field data and SQL statement generation rules, and splice them with the SQL statement framework to obtain SQL query statements;

[0107] The splicing module is also used to generate a time processing SQL statement fragment according to the window calculation mode, the window time and the SQL statement generation rule, and splice it with the SQL query statement to obtain an SQL query statement after splicing the time processing SQL statement fragment;

[0108] The splicing module is also used to generate and process SQL statements according to the aggregation condition parameters and SQL statement generation rules, and splice them with the SQL query statement after splicing the SQL statement fragments to obtain the SQL statement.

[0109] In some embodiments, the configuration information further includes a project field. After sending the SQL statement to the computing engine to generate the real-time analysis task, the apparatus 400 for generating the real-time analysis task may further include:

[0110] The splicing module is also used to splice project fields, product fields and window time to generate storage table names;

[0111] The storage module is used to create a database table according to the preset database type and database address corresponding to the storage table name and project field, so as to store the execution results of the real-time analysis task.

[0112] In some embodiments, the generation module 404 is further used to send the SQL statement to the computing engine corresponding to the storage group for execution according to the relationship between the storage group of the data stream to be analyzed and the computing engine, so as to generate a real-time analysis task.

[0113] Figure 4 The various modules in the device shown can be implemented Figure 1 Each step in the above steps can achieve the corresponding technical effect, which will not be repeated here for the sake of brevity.

[0114] Figure 5 A schematic diagram of the hardware structure of a terminal device provided in an embodiment of the present application is shown.

[0115] The terminal device may include a processor 501 and a memory 502 storing computer program instructions.

[0116] Specifically, the processor 501 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0117] The memory 502 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 502 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In one example, the memory 502 may include a removable or non-removable (or fixed) medium, or the memory 502 is a non-volatile solid-state memory. The memory 502 may be inside or outside the integrated gateway disaster recovery device.

[0118] In one example, the memory 502 may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk storage medium device, an optical storage medium device, a flash memory device, an electrical, optical or other physical / tangible memory storage device. Thus, typically, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method for real-time analysis task generation according to the present disclosure.

[0119] The processor 501 reads and executes the computer program instructions stored in the memory 502 to implement Figure 1 The method for generating real-time analysis tasks in the illustrated embodiment.

[0120] In one example, the terminal device may further include a communication interface 503 and a bus 504. Figure 5 As shown, the processor 501, the memory 502, and the communication interface 503 are connected via a bus 504 and communicate with each other.

[0121] The communication interface 503 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0122] Bus 504 includes hardware, software or both, and the components of terminal equipment are coupled to each other. For example, but not limitation, bus may include accelerated graphics port (Accelerated Graphics Port, AGP) or other graphics bus, enhanced industry standard architecture (Extended Industry Standard Architecture, EISA) bus, front side bus (Front Side Bus, FSB), Hyper Transport (Hyper Transport, HT) interconnection, industry standard architecture (Industry Standard Architecture, ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 504 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the application considers any suitable bus or interconnection.

[0123] In addition, in combination with the method for generating a real-time analysis task in the above embodiment, the embodiment of the present application may provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when the computer program instructions are executed by a processor, any one of the methods for generating a real-time analysis task in the above embodiment is implemented.

[0124] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed, implements any one of the methods for generating real-time analysis tasks in the above embodiments.

[0125] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0126] The functional blocks shown in the structural block diagram described above can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), appropriate firmware, plug-in, function card, etc. When implemented in software, the elements of the present application are programs or text segments used to perform the required tasks. The program or text segment can be stored in a machine-readable medium, or transmitted on a transmission medium or communication link via a data signal carried in a carrier. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, read-only memory (Read-Only Memory, ROM), flash memory, erasable read-only memory (Erasable ReadOnly Memory, EROM), floppy disks, compact disc read-only memory (Compact Disc Read-Only Memory, CD-ROM), optical discs, hard disks, optical fiber media, radio frequency (Radio Frequency, RF) links, etc. The text segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0127] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.

[0128] Aspects of the present disclosure are described above with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the function / action specified in one or more boxes of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field programmable logic circuit. It can also be understood that each box in the block diagram and / or flowchart and the combination of boxes in the block diagram and / or flowchart can also be implemented by dedicated hardware that performs a specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0129] The above is only a specific implementation of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the protection scope of this application.

Claims

1. A method for generating a real-time analysis task, characterized in that: include: Acquire the data flow to be analyzed and configuration information, wherein the configuration information includes input information, processing information and output information, wherein the input information includes an outer layer architecture and an inner layer architecture, the processing information includes aggregation condition parameters, and the output information includes a dynamic output field; Determine the outer architecture of the data stream to be analyzed according to the relationship between the message type of the data stream and the outer architecture, and parse the product fields and dynamic field sets in the data stream to be analyzed based on the outer architecture; Determine the inner layer architecture according to the relationship between the product field and the inner layer architecture, and parse the dynamic output field data in the dynamic field set based on the inner layer architecture, the dynamic field set and the dynamic output field; Determine the SQL statement based on the inner layer architecture, outer layer architecture, product fields, dynamic output field data, aggregation condition parameters and preset structured query SQL statement generation rules; The SQL statement is sent to the computing engine for execution to generate a real-time analysis task.

2. The method for generating a real-time analysis task according to claim 1, characterized in that: The step of obtaining the data stream to be analyzed includes: Get the data source address of the data stream to be analyzed; Select the target connector based on the relationship between the data source address and the preset connector; The data stream to be analyzed is obtained from a data source corresponding to a data source address of the data stream to be analyzed based on the target connector.

3. The method for generating a real-time analysis task according to claim 1, characterized in that: Get configuration information, including: Obtain input information, processing information, and output information identifiers selected by the user in the visual interface; The corresponding input information, processing information and output information are obtained according to the input information, processing information and output information identifiers.

4. The method for generating a real-time analysis task according to claim 1, characterized in that: The configuration information also includes a window calculation mode and a window time. The SQL statement is determined based on the inner layer architecture, the outer layer architecture, the product field, the dynamic output field data, the aggregation condition parameters and the preset structured query SQL statement generation rules, including: Build the SQL statement framework based on the outer layer architecture, dynamic field collection, product fields and SQL statement generation rules; Generate inner-layer SQL statement fragments according to the inner-layer architecture, dynamic output field data and SQL statement generation rules, and splice them with the SQL statement framework to obtain SQL query statements; Generate a time processing SQL statement fragment according to the window calculation mode, the window time, and the SQL statement generation rule, and splice it with the SQL query statement to obtain an SQL query statement after splicing the time processing SQL statement fragment; A processing SQL statement is generated according to the aggregation condition parameters and the SQL statement generation rule, and is spliced ​​with the SQL query statement after splicing the time processing SQL statement fragments to obtain an SQL statement.

5. The method for generating a real-time analysis task according to claim 4, characterized in that: The configuration information also includes a project field. After sending the SQL statement to the computing engine to generate a real-time analysis task, the method further includes: Concatenate the project field, product field and window time to generate the storage table name; Create a database table based on the preset database type and database address corresponding to the storage table name and project field to store the execution results of the real-time analysis task.

6. The method for generating a real-time analysis task according to claim 1, characterized in that: The data stream to be analyzed is grouped and stored according to preset rules, and the SQL statement is sent to the computing engine for execution to generate a real-time analysis task, including: According to the relationship between the storage group and the computing engine of the data stream to be analyzed, the SQL statement is sent to the computing engine corresponding to the storage group for execution, generating a real-time analysis task.

7. A device for real-time analysis task generation, characterized in that: The device comprises: An acquisition module, used to acquire the data flow to be analyzed and configuration information, wherein the configuration information includes input information, processing information and output information, wherein the input information includes an outer layer structure and an inner layer structure, the processing information includes aggregation condition parameters, and the output information includes a dynamic output field; A parsing module, used to determine the outer layer architecture of the data stream to be analyzed according to the relationship between the message type of the data stream and the outer layer architecture, and parse out the product fields and dynamic field sets in the data stream to be analyzed based on the outer layer architecture; The parsing module is further used to determine the inner layer architecture according to the relationship between the product field and the inner layer architecture, and parse the dynamic output field data in the dynamic field set based on the inner layer architecture, the dynamic field set and the dynamic output field; A determination module, used to determine the SQL statement based on the inner layer architecture, the outer layer architecture, the product field, the dynamic output field data, the aggregation condition parameters and the preset structured query SQL statement generation rules; The generation module is used to send the SQL statement to the computing engine for execution and generate a real-time analysis task.

8. A terminal device, characterized in that: The device comprises: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, the method for generating a real-time analysis task as described in any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by a processor, the method for generating a real-time analysis task as described in any one of claims 1 to 6 is implemented.

10. A computer program product, characterized in that When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the method for generating a real-time analysis task as described in any one of claims 1 to 6.

Images