Data processing method and device, electronic equipment and storage medium

By activating the corresponding Source components and Sink components according to the category relationship between the target data providing end and the receiving end in the Flume system, the problem of poor applicability of data processing methods in the prior art is solved, and real-time data from multiple data providing ends and receiving ends is realized Processing ensures the real-time and applicability of the data.

CN119003636APending Publication Date: 2024-11-22CHINA CONSTRUCTION BANK +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411086007.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-08
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

The existing data processing methods cannot be applied to multiple data providers and multiple data receiving ends, and there is a problem of poor applicability.

Method used

By using multiple candidate collector Source components, channel components and candidate receiver Sink components in the Flume system, the corresponding components are activated according to the category relationship between the target data provisioning end and the receiving end, and real-time data acquisition and storage are realized.

Benefits of technology

Real-time data acquisition for multiple data providers and real-time data storage for multiple data receivers is realized. It is highly applicable, and the Flume system has the advantages of lightweight deployment, ensuring real-time data processing.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119003636A_ABST
    Figure CN119003636A_ABST
Patent Text Reader

Abstract

The invention relates to a data processing method and device, electronic equipment and a storage medium, and belongs to the technical field of financial science and technology, information technology industry and data processing. The method comprises the following steps: acquiring a candidate Source component having a corresponding relationship with the category of a target data providing end, taking the candidate Source component as a target Source component, and activating the target Source component; obtaining a candidate Sink component having a corresponding relationship with the category of the target data receiving end as a target Sink component, and activating the target Sink component; obtaining a target data set from the target data providing end through the target Source component according to a data providing mode of the target data providing end; and storing the second data set to the target data receiving end through the target Sink component according to the data storage mode of the target data receiving end. Therefore, the method is suitable for real-time data acquisition of various data providing ends and real-time data storage of various data receiving ends, and is good in applicability.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the fields of financial technology, information technology industry, and data processing technology, and in particular to a data processing method, device, electronic device, computer-readable storage medium, and computer program product. Background Art

[0002] At present, with the continuous development of big data technology, data processing has been widely used in finance, autonomous driving, smart home appliances and other fields, and there are many types of data providers and data receivers in various fields. However, the data processing methods in related technologies cannot be applied to multiple data providers and multiple data receivers, and have the problem of poor applicability. Summary of the invention

[0003] The present disclosure provides a data processing method, device, electronic device, computer-readable storage medium and computer program product to at least solve the problem that the data processing method in the related art cannot be applied to a variety of data providing terminals and a variety of data receiving terminals and has poor applicability. The technical solution of the present disclosure is as follows:

[0004] According to a first aspect of an embodiment of the present disclosure, a data processing method is provided, wherein a Flume system includes a plurality of candidate collector Source components, a channel Channel component, and a plurality of candidate receiver Sink components, wherein there is a corresponding relationship between the candidate Source components and the categories of the candidate data providers, there is a corresponding relationship between the candidate Sink components and the categories of the candidate data receivers, and the Channel component is in an activated state;

[0005] The method includes: obtaining a candidate Source component that has a corresponding relationship with the category of the target data provider as the target Source component, and activating the target Source component; obtaining a candidate Sink component that has a corresponding relationship with the category of the target data receiving end as the target Sink component, and activating the target Sink component; obtaining a target data set from the target data provider according to the data providing method of the target data provider through the target Source component; storing a first data set to be stored through the Channel component, wherein the first data set is obtained by processing a target data set from at least one target data provider; obtaining a second data set to be stored in the target data receiving end from the first data set through the target Sink component, wherein the second data set is a subset of the first data set; and storing the second data set in the target data receiving end through the target Sink component according to the data storage method of the target data receiving end.

[0006] According to a second aspect of an embodiment of the present disclosure, a data processing device is provided, wherein a Flume system includes a plurality of candidate collector Source components, a channel Channel component, and a plurality of candidate receiver Sink components, wherein there is a corresponding relationship between the candidate Source components and the categories of the candidate data providing terminals, there is a corresponding relationship between the candidate Sink components and the categories of the candidate data receiving terminals, and the Channel component is in an activated state;

[0007] The device includes: a first activation module, configured to obtain a candidate Source component corresponding to a category of a target data provider as a target Source component, and activate the target Source component; a second activation module, configured to obtain a candidate Sink component corresponding to a category of a target data receiving end as a target Sink component, and activate the target Sink component; a first acquisition module, configured to obtain a target data set from the target data provider according to a data providing method of the target data provider through the target Source component; a first storage module, configured to store a first data set to be stored through the Channel component, wherein the first data set is obtained by processing a target data set from at least one target data provider; a second acquisition module, configured to obtain a second data set to be stored in the target data receiving end from the first data set through the target Sink component, wherein the second data set is a subset of the first data set; and a second storage module, configured to store the second data set in the target data receiving end through the target Sink component according to the data storage method of the target data receiving end.

[0008] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the data processing method as described in the first aspect.

[0009] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device can execute the data processing method as described in the first aspect.

[0010] According to a fifth aspect of an embodiment of the present disclosure, there is provided a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the data processing method as described in the first aspect is implemented.

[0011] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects: multiple candidate Source components can be pre-developed considering the categories of candidate data providers, which are suitable for real-time data acquisition of various data providers, and multiple candidate Sink components can be pre-developed considering the categories of candidate data receiving terminals, which are suitable for real-time data storage of various data receiving terminals, and have strong applicability. The target Source component and the target Sink component can be determined therefrom, and the target data set can be obtained from the data provider according to the data providing method of the target data provider through the target Source component, and the second data set can be stored in the data receiving terminal according to the data storage method of the target data receiving terminal through the target Sink component. The Flume system has the advantage of lightweight deployment, which ensures real-time data processing.

[0012] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0014] Figure 1 It is a flowchart of a data processing method according to the first embodiment of the present disclosure.

[0015] Figure 2 It is a flowchart of a data processing method according to the second embodiment of the present disclosure.

[0016] Figure 3 It is a flowchart of a data processing method according to the third embodiment of the present disclosure.

[0017] Figure 4 is a block diagram of a data processing device according to a first embodiment of the present disclosure.

[0018] Figure 5 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0019] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0020] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0021] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this disclosure are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards.

[0022] It should be noted that the acquisition, transmission, storage, use, and processing of data in the technical solution of the present disclosure are in compliance with the relevant provisions of relevant laws and regulations. In the embodiments of the present disclosure, certain software, components, models, and other existing solutions in the industry may be mentioned, and they should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present disclosure, but it does not mean that the applicant has or will necessarily use the solution.

[0023] Figure 1 It is a flowchart of a data processing method according to the first embodiment of the present disclosure.

[0024] like Figure 1 As shown, the data processing method of the first embodiment of the present disclosure includes the following steps:

[0025] S101, obtaining a candidate Source component corresponding to the category of the target data provider as a target Source component, and activating the target Source component.

[0026] It should be noted that the execution subject of the data processing method of the present disclosure is an electronic device, wherein the electronic device may include a mobile phone, a notebook, a vehicle-mounted device, a smart home appliance, etc. The data processing method of the embodiment of the present disclosure may be executed by the data processing device of the embodiment of the present disclosure, and the data processing device of the embodiment of the present disclosure may be configured in any electronic device to execute the data processing method of the embodiment of the present disclosure.

[0027] In the embodiments of the present disclosure, the Flume system includes multiple candidate Source (collector) components, Channel (channel) components and multiple Sink (receiver) components. Among them, there is a corresponding relationship between the candidate Source component and the category of the candidate data provider, there is a corresponding relationship between the candidate Sink component and the category of the candidate data receiving end, and the Channel component is in an activated state. It should be noted that the Flume system is a distributed, highly reliable, and highly available data collection, aggregation, and transmission system.

[0028] It should be noted that the candidate Source component is used to obtain data from the candidate data provider that has a corresponding relationship with itself according to the data providing method of the candidate data provider that has a corresponding relationship with itself, and the candidate Sink component is used to store data to the candidate data receiving end that has a corresponding relationship with itself according to the data storage method of the candidate data receiving end that has a corresponding relationship with itself.

[0029] It should be noted that the terms data provider, data source, etc. are interchangeable. There are no excessive restrictions on data providers, for example, they may include databases (such as relational databases, non-relational databases), caches (such as memory), files (such as log files, server files, local files), message queues, data interfaces, data stream processing systems, sensors, IoT devices, etc.

[0030] It should be noted that the terms data receiver, data consumer, data destination, etc. are interchangeable. There is no excessive limitation on the data receiver, for example, it may include database, cache, file, message queue, data interface, data stream processing system, IoT device, etc.

[0031] It should be noted that there are no excessive restrictions on the number of candidate Source components and candidate Sink components included in the Flume system. It is understandable that different categories of candidate data providers may correspond to different candidate Source components, and different categories of candidate data receiving terminals may correspond to different candidate Sink components. For example, there is a one-to-one correspondence between the candidate Source components and the categories of the candidate data providers, and there is a one-to-one correspondence between the candidate Sink components and the categories of the candidate data receiving terminals. The number of target data providers is at least one, and multiple candidate data providers include a target data provider, and multiple candidate Source components include a target Source component.

[0032] For example, there is a mapping relationship between candidate Source component 1 and the category of candidate data provider A, there is a mapping relationship between candidate Source component 2 and the category of candidate data provider B, and there is a mapping relationship between candidate Source component 3 and the category of candidate data provider C.

[0033] If the target data provider includes the candidate data provider A, a candidate Source component 1 corresponding to the category of the candidate data provider A may be obtained as the target Source component, and the candidate Source component 1 may be activated.

[0034] If the target data provider includes a candidate data provider B, a candidate Source component 2 having a corresponding relationship with the category of the candidate data provider B may be obtained as the target Source component, and the candidate Source component 2 may be activated.

[0035] It should be noted that the activation of the target Source component can be achieved by using any component activation method in the relevant technology, and no excessive restrictions are made here.

[0036] S102, obtaining a candidate Sink component that has a corresponding relationship with the category of the target data receiving end as the target Sink component, and activating the target Sink component.

[0037] It should be noted that the number of target data receiving ends is at least one, the multiple candidate data receiving ends include the target data receiving end, and the multiple candidate Sink components include the target Sink component.

[0038] For example, there is a mapping relationship between the categories of candidate Sink component 1 and candidate data receiving terminal D, there is a mapping relationship between the categories of candidate Sink component 2 and candidate data receiving terminal E, and there is a mapping relationship between the categories of candidate Sink component 3 and candidate data receiving terminal F.

[0039] If the target data receiving end includes a candidate data receiving end D, a candidate Sink component 1 corresponding to the category of the candidate data receiving end D can be obtained as the target Sink component, and the candidate Sink component 1 can be activated.

[0040] If the target data receiving end includes a candidate data receiving end E, a candidate Sink component 2 corresponding to the category of the candidate data receiving end E can be obtained as the target Sink component, and the candidate Sink component 2 can be activated.

[0041] It should be noted that the activation of the target Sink component can be achieved by using any component activation method in the relevant technology, and no excessive restrictions are made here.

[0042] S103, obtaining a target data set from a target data provider through a target Source component according to a data providing mode of the target data provider.

[0043] It is understandable that the number of target data sets from the same target data provider is at least one, and the target data sets from different target data providers may be different. Different target data providers may have different data providing methods, and the number of data providing methods for the same target data provider is at least one.

[0044] In one embodiment, a target data set is obtained from a target data provider through a target Source component in accordance with a data providing method of the target data provider, including obtaining a first target code through a target Source component, wherein the first target code is a pre-set code for data acquisition, and the first target code is obtained based on the data providing method of the target data provider, the first target code is executed through the target Source component, and the target data set is obtained from the target data provider through the first target code in accordance with the data providing method of the target data provider.

[0045] It is understandable that the developer can customize and develop the first target code based on the data providing mode of the target data providing end, so that the first target code is suitable for data acquisition of the target data providing end. There is no excessive limitation on the first target code. For example, it can be written in any programming language in the relevant technology.

[0046] In one embodiment, obtaining the target data set from the target data provider according to the data providing mode of the target data provider by the target Source component includes identifying, by the target Source component, that the data providing mode of the target data provider is to collect data from the target data provider, and the Source component collecting the target data set from the target data provider. Thus, when it is identified that the data providing mode of the data provider is to collect data, the target data set can be collected from the data provider by the Source component.

[0047] In one embodiment, obtaining the target data set from the target data provider according to the data providing mode of the target data provider through the target Source component includes identifying through the target Source component that the data providing mode of the data provider is to receive data sent by the data provider, and receiving through the target Source component the target data set sent by the data provider. Thus, when it is identified that the data providing mode of the data provider is to receive data, the target data set sent by the data provider can be received through the Source component.

[0048] It should be noted that there are not too many restrictions on the transmission method of the target data set. For example, it can include log push, TCP (Transmission Control Protocol) push, UDP (User Datagram Protocol) push, streaming push, etc.

[0049] In one embodiment, the target data set is obtained from the target data provider by the target Source component according to the data providing mode of the target data provider, including if the target data provider has multiple data providing modes, the target data set is obtained from the target data provider by the target Source component according to the data providing mode with the highest priority of the target data provider. It can be understood that in this embodiment, the data providing mode has a priority, and the priorities of different data providing modes may be different. Therefore, when the data providing mode of the data provider has multiple data providing modes, the target data set can be obtained from the data provider by the Source component according to the data providing mode with the highest priority of the data provider.

[0050] S104: Store a first data set to be stored through a Channel component, wherein the first data set is obtained by processing a target data set from at least one target data provider.

[0051] It should be noted that there are no excessive restrictions on the processing methods of the target data set, for example, it may include data analysis, data processing, etc. Among them, data analysis may include data analysis through delimiters, extraction of labels or key-value pairs, text analysis based on NLP (Natural Language Processing), structured processing, etc., and data processing may include data cleaning, data conversion, feature engineering, data aggregation, data segmentation, quality inspection, etc.

[0052] It should be noted that the Channel component can store the first data set to be stored by using any data storage method of a Channel component in the relevant technology, and the Channel component can use any Channel component in the relevant technology, and no excessive restrictions are made here.

[0053] S105, obtaining a second data set to be stored in a target data receiving end from the first data set through a target Sink component, wherein the second data set is a subset of the first data set.

[0054] It can be understood that the number of the second data sets to be stored in the same target data receiving end is at least one, and the second data sets to be stored in different target data receiving ends may be different.

[0055] It should be noted that obtaining the second data set to be stored in the target data receiving end from the first data set by the target Sink component can be implemented by using the data processing method of any Sink component in the relevant technology, and no excessive restrictions are made here.

[0056] In one embodiment, obtaining a second data set to be stored in a target data receiving end from a first data set by a target Sink component includes filtering the second data set from the first data set based on a data storage requirement of the target data receiving end by the target Sink component. It should be noted that there are no excessive restrictions on the data storage requirement, for example, it may include the data category, data format, data volume, data collection time period, etc. to be stored in the target data receiving end.

[0057] S106: Store the second data set to the target data receiving end through the target Sink component according to the data storage method of the target data receiving end.

[0058] It is understandable that different target data receiving ends may have different data storage modes, and the number of data storage modes for the same target data receiving end is at least one.

[0059] In one embodiment, the second data set is stored in the target data receiving end through the target Sink component according to the data storage method of the target data receiving end, including obtaining a second target code through the target Sink component, wherein the second target code is a pre-set code for data storage, and the second target code is obtained based on the data storage method of the target data receiving end, executing the second target code through the target Sink component, and storing the second data set in the target data receiving end through the second target code according to the data storage method of the target data receiving end.

[0060] It is understandable that the developer can customize and develop the second target code based on the data storage method of the target data receiving end, so that the second target code is suitable for the data storage of the target data receiving end. There is no excessive limitation on the second target code. For example, it can be written in any programming language in the relevant technology.

[0061] In one embodiment, the second data set is stored in the target data receiving terminal by the target Sink component according to the data storage mode of the target data receiving terminal, including if the target data receiving terminal has multiple data storage modes, the second data set is stored in the target data receiving terminal by the target Sink component according to the data storage mode with the highest priority of the target data receiving terminal. It can be understood that the data storage mode in this embodiment has a priority, and the priorities of different data storage modes may be different. Thus, when there are multiple data storage modes at the data receiving terminal, the second data set can be stored in the target data receiving terminal by the Sink component according to the data storage mode with the highest priority of the data receiving terminal.

[0062] In summary, according to the data processing method provided by the embodiment of the present disclosure, a candidate Source component corresponding to the category of the target data provider is obtained as the target Source component, and the target Source component is activated. A candidate Sink component corresponding to the category of the target data receiving end is obtained as the target Sink component, and the target Sink component is activated. The target data set is obtained from the target data provider according to the data providing method of the target data provider through the target Source component, the first data set to be stored is stored through the Channel component, the second data set to be stored in the target data receiving end is obtained from the first data set through the target Sink component, and the second data set is stored in the target data receiving end through the target Sink component according to the data storage method of the target data receiving end. Therefore, multiple candidate Source components can be developed in advance considering the categories of candidate data providers, which are suitable for real-time data acquisition of various data providers, and multiple candidate Sink components can be developed in advance considering the categories of candidate data receiving terminals, which are suitable for real-time data storage of various data receiving terminals, with strong applicability, and the target Source component and the target Sink component can be determined therefrom. The target data set can be obtained from the data provider through the target Source component according to the data providing method of the target data provider, and the second data set can be stored in the data receiving terminal through the target Sink component according to the data storage method of the target data receiving terminal. The Flume system has the advantage of lightweight deployment, which ensures real-time data processing.

[0063] Figure 2 It is a flowchart of a data processing method according to the second embodiment of the present disclosure.

[0064] like Figure 2 As shown, the data processing method of the second embodiment of the present disclosure includes the following steps:

[0065] S201, obtaining a candidate Source component that has a corresponding relationship with the category of the target data provider as a target Source component, and activating the target Source component.

[0066] S202, obtain a candidate Sink component that has a corresponding relationship with the category of the target data receiving end as the target Sink component, and activate the target Sink component.

[0067] S203, obtaining the target data set from the target data provider through the target Source component according to the data providing mode of the target data provider.

[0068] For the relevant contents of steps S201 - S203 , please refer to the above embodiments and will not be described again here.

[0069] S204, filtering the target data set through the target Source component to obtain a third data set.

[0070] It should be noted that there is a corresponding relationship between the target data set and the third data set, for example, the target data set and the third data set correspond one to one. For example, the third data set is a subset of the target data set.

[0071] In one embodiment, the target data set is filtered by the target Source component to obtain the third data set, including obtaining a data filtering code by the target Source component, wherein the data filtering code is a pre-set code for data filtering, executing the data filtering code by the target Source component, and filtering the target data set by the data filtering code to obtain the third data set. Thus, the data filtering code can be obtained and executed by the Source component to implement data filtering.

[0072] It should be noted that there is no excessive limitation on the data screening code. For example, the data screening code can be written in any programming language in the relevant technology, for example, it can include Morphline code. It should be noted that Morphline is a configuration language and execution framework for writing data processing processes.

[0073] In one embodiment, the target data set is screened by the target Source component to obtain the third data set, including obtaining the data screening rule by the target Source component, filtering out the first category of data that does not meet the data screening rule from the target data set by the target Source component, and retaining the second category of data that meets the data screening rule to update the target data set, and using the updated target data set as the third data set by the target Source component. Thus, the first category of data can be filtered out from the target data set by the Source component taking into account the data screening rule, and the second category of data can be retained to update the target data set, and the updated target data set can be used as the third data set to achieve data screening.

[0074] It should be noted that the data in the third data set meets the data screening rule, and the third data set only includes the second type of data, and does not include the first type of data. There are no excessive restrictions on the data screening rules, for example, they may include the data category to be screened, the data format, the data volume, the data collection time period, etc.

[0075] In some examples, obtaining data screening rules through the target Source component includes obtaining the user's data display requirements through the target Source component, and generating data screening rules based on the data display requirements through the target Source component. Thus, the data screening rules can be automatically generated by the Source component taking into account the user's data display requirements, and personalized generation of data screening rules can be achieved.

[0076] It should be noted that there are not too many restrictions on data display requirements. For example, it may include the data category to be displayed, data format, data volume, data collection time period, etc.

[0077] For example, data filtering rules are generated based on data display requirements through the target Source component, including combining multiple data display requirements through the target Source component to obtain data filtering rules.

[0078] For example, data filtering rules are generated based on data display requirements through the target Source component, including obtaining a template of the data filtering rules through the target Source component, and replacing the text to be replaced in the template of the data filtering rules with the data display requirements through the target Source component to obtain the data filtering rules.

[0079] S205: Process the third data set from at least one target data provider through a filter to obtain a first data set.

[0080] In this embodiment, the Flume system also includes filters, such as Morphline filters. There are no excessive restrictions on the specific methods of data processing, such as data cleaning, data conversion, feature engineering, data aggregation, data sharding, quality inspection, etc.

[0081] For example, if the target data provider includes candidate data providers A and B, a third data set from candidate data provider A and a third data set from candidate data provider B are processed by a filter to obtain a first data set.

[0082] S206: Store the first data set to be stored through the Channel component, wherein the first data set is obtained by processing a target data set from at least one target data provider.

[0083] S207, obtaining a second data set to be stored in a target data receiving end from the first data set through the target Sink component, wherein the second data set is a subset of the first data set.

[0084] S208: Store the second data set to the target data receiving end through the target Sink component according to the data storage method of the target data receiving end.

[0085] For the relevant contents of steps S206-S208, please refer to the above embodiment and will not be repeated here.

[0086] In summary, according to the data processing method provided by the embodiment of the present disclosure, the target data set is screened by the target Source component to obtain a third data set, and the third data set from at least one target data provider is processed by the filter to obtain a first data set. Thus, the acquired target data set can be screened by the Source component, which helps to improve data quality, reduce the amount of data to be processed, and improve data processing efficiency, and the third data set from at least one data provider is processed by the filter to obtain the first data set to be stored.

[0087] Figure 3 It is a flowchart of a data processing method according to the third embodiment of the present disclosure.

[0088] like Figure 3 As shown, the data processing method of the third embodiment of the present disclosure processes the target data set to obtain a first data set, including the following steps:

[0089] S301, obtaining data filtering rules through the target Source component.

[0090] S302, filtering out the first type of data that does not satisfy the data screening rule from the target data set through the target Source component, and retaining the second type of data that satisfies the data screening rule, so as to update the target data set.

[0091] S303: Using the target Source component, the updated target data set is used as the third data set.

[0092] For the relevant contents of steps S301 - S303 , please refer to the above embodiment and will not be described again here.

[0093] S304, sending the target indication information to the target data provider of the first category data source through the target Source component, wherein the target indication information is used to indicate that the first category data does not meet the data screening rules, and / or is used to indicate that the first category data is to be corrected.

[0094] S305, receiving, through the target Source component, first corrected data sent by a target data provider of the first type of data source, wherein the first corrected data is obtained by correcting the first type of data.

[0095] S306, identifying that the first revised data meets the data screening rule through the target Source component, and adding the first revised data to the third data set.

[0096] S307, identifying through the target Source component that the first revised data does not satisfy the data screening rule, and updating the first revised data to the first category of data.

[0097] It should be noted that after step S307, the process may return to step S304 and subsequent steps. In this embodiment, the third data set also includes first modified data, which refers to data obtained from the target data provider and obtained by modifying the first type of data.

[0098] For example, if the target data provider includes a candidate data provider A, the target data set X is obtained from the candidate data provider A through the target Source component.

[0099] The target data set X includes the first type of data X1 that does not meet the data screening rule and the second type of data X2 that meets the data screening rule. The first type of data X1 is filtered out from the target data set X through the target Source component, and the second type of data X2 is retained to update the target data set X. The updated target data set X is used as the third data set X3 through the target Source component. The third data set X3 only includes the second type of data X2.

[0100] The target indication information 1 is sent to the candidate data provider A of the source of the first category data X1 through the target Source component, wherein the target indication information 1 is used to indicate that the first category data X1 does not meet the data screening rules, and / or to indicate that the first category data X1 should be corrected.

[0101] The first corrected data X4 sent by the candidate data provider A which is the source of the first type of data X1 is received through the target Source component, wherein the first corrected data X4 is obtained by correcting the first type of data X1.

[0102] The target Source component identifies that the first revised data X4 satisfies the data screening rule, and adds the first revised data X4 to the third data set X3.

[0103] Alternatively, the target Source component identifies that the first corrected data X4 does not meet the data screening rules, updates the first corrected data X4 to the first category data X1, and returns to execute the target indication information 1 sent to the candidate data provider A of the source of the first category data X1 through the target Source component and its subsequent steps.

[0104] For example, if the target data provider includes the candidate data provider B, the target data set Y is obtained from the candidate data provider B through the target Source component.

[0105] The target data set Y includes the first type of data Y1 that does not meet the data screening rule and the second type of data Y2 that meets the data screening rule. The first type of data Y1 is filtered out from the target data set Y through the target Source component, and the second type of data Y2 is retained to update the target data set Y. The updated target data set Y is used as the third data set Y3 through the target Source component. The third data set Y3 only includes the second type of data Y2.

[0106] The target indication information 2 is sent to the candidate data provider B of the first category data Y1 through the target Source component, wherein the target indication information 2 is used to indicate that the first category data Y1 does not meet the data screening rules, and / or to indicate that the first category data Y1 should be corrected.

[0107] The first corrected data Y4 sent by the candidate data provider B which is the source of the first type of data Y1 is received through the target Source component, wherein the first corrected data Y4 is obtained by correcting the first type of data Y1.

[0108] The target Source component identifies that the first revised data Y4 satisfies the data screening rule, and adds the first revised data Y4 to the third data set Y3.

[0109] Alternatively, the target Source component identifies that the first corrected data Y4 does not meet the data screening rules, updates the first corrected data Y4 to the first category data Y1, and returns to execute the target indication information 2 sent to the candidate data provider B of the first category data Y1 source through the target Source component and its subsequent steps.

[0110] S308: Process the third data set from at least one target data provider through a filter to obtain a first data set.

[0111] For the relevant contents of step S308, please refer to the above embodiment, which will not be repeated here.

[0112] In summary, according to the data processing method provided by the embodiment of the present disclosure, the target indication information is sent to the target data provider of the first category data source through the target Source component, wherein the target indication information is used to indicate that the first category data does not meet the data screening rules, and / or is used to indicate that the first category data is to be corrected, and the first corrected data sent by the target data provider of the first category data source is received through the target Source component, wherein the first corrected data is obtained by correcting the first category data, and the target Source component is used to identify that the first corrected data meets the data screening rules, and the first corrected data is added to the third data set, or the target Source component is used to identify that the first corrected data does not meet the data screening rules, and the first corrected data is updated to the first category data, and the target indication information is sent to the target data provider of the first category data source through the target Source component and its subsequent steps are returned to execute. Therefore, the target indication information can be sent to the data provider of the first category data source through the Source component, so as to promptly inform the data provider that the first category data does not meet the data screening rules, or promptly notify the data provider to correct the first category data, receive the first corrected data sent by the data provider through the Source component, and when the first corrected data meets the data screening rules, add the first corrected data to the third data set, and when the first corrected data does not meet the data screening rules, update the first corrected data to the first category data, and continue to execute the steps of sending the target indication information to the data provider of the first category data source and its subsequent steps, so as to realize multiple data corrections until the data screening rules are met.

[0113] Based on any of the above embodiments, the method further includes correcting the first type of data according to the data screening rules through the target Source component to obtain second corrected data that meets the data screening rules, and adding the second corrected data to the third data set through the target Source component. Thus, the Source component can automatically correct the data according to the data screening rules to obtain the second corrected data that meets the data screening rules, and add the second corrected data to the third data set.

[0114] It should be noted that, in this embodiment, the third data set also includes second corrected data, and the second corrected data refers to data obtained by the Source component performing correction processing on the first type of data.

[0115] For example, continuing to take the first category of data X1 as an example, the first category of data X1 is corrected according to the data filtering rules through the target Source component to obtain the second corrected data X5 that meets the data filtering rules, and the second corrected data X5 is added to the third data set X3 through the target Source component.

[0116] For example, continuing to take the first category of data Y1 as an example, the first category of data Y1 is corrected according to the data filtering rules through the target Source component to obtain the second corrected data Y5 that meets the data filtering rules, and the second corrected data Y5 is added to the third data set Y3 through the target Source component.

[0117] Based on any of the above embodiments, the method further includes obtaining a fourth data set to be displayed by a target display device based on a second data set from at least one target data receiving terminal, and controlling the target display device to display data in the fourth data set in a jumping or scrolling manner. Thus, the second data set from at least one data receiving terminal can be considered to obtain a fourth data set to be displayed by a display device, and the display device can be controlled to display data in the fourth data set in a jumping or scrolling manner, thereby achieving a display effect of data jumping or data scrolling.

[0118] It should be noted that the target display device is not limited in many ways, for example, it may include a display screen, a projection device, etc. The number of fourth data sets to be displayed by the same target display device is at least one, and different target display devices may have different fourth data sets to be displayed.

[0119] It should be noted that the data in the fourth data set can be displayed in a jumping manner by any jumping display method in the relevant technology, and the data in the fourth data set can be displayed in a scrolling manner by any scrolling display method in the relevant technology, and no further limitation is given here. The jumping display method and the scrolling display method may include sequence control, flow rate control, and slicing control.

[0120] In one embodiment, obtaining a fourth data set to be displayed by a target display device based on a second data set from at least one target data receiving terminal includes processing the second data set from at least one target data receiving terminal to obtain the fourth data set. It should be noted that the relevant content of the data processing can be referred to the above embodiment, and will not be repeated here.

[0121] Figure 4 The block diagram of the data processing device according to the first embodiment of the present disclosure. The Flume system includes multiple candidate collector Source components, channel Channel components and multiple candidate receiver Sink components, wherein there is a corresponding relationship between the candidate Source components and the categories of the candidate data providers, there is a corresponding relationship between the candidate Sink components and the categories of the candidate data receivers, and the Channel component is in an activated state.

[0122] like Figure 4 As shown, the data processing device 400 of the embodiment of the present disclosure includes: a first activation module 401, a second activation module 402, a first acquisition module 403, a first storage module 404, a second acquisition module 405 and a second storage module 406.

[0123] The first activation module 401 is configured to obtain a candidate Source component corresponding to the category of the target data provider as a target Source component, and activate the target Source component;

[0124] The second activation module 402 is configured to obtain a candidate Sink component having a corresponding relationship with the category of the target data receiving end as a target Sink component, and activate the target Sink component;

[0125] The first acquisition module 403 is configured to acquire a target data set from the target data provider through the target Source component according to the data providing mode of the target data provider;

[0126] A first storage module 404 is configured to store a first data set to be stored through the Channel component, wherein the first data set is obtained by processing a target data set from at least one target data provider;

[0127] A second acquisition module 405 is configured to acquire a second data set to be stored in the target data receiving end from the first data set through the target Sink component, wherein the second data set is a subset of the first data set;

[0128] The second storage module 406 is configured to store the second data set to the target data receiving end through the target Sink component according to the data storage method of the target data receiving end.

[0129] In one embodiment of the present disclosure, the first acquisition module 403 is further configured to: identify, through the target Source component, that the data providing method of the target data provider is to collect data from the target data provider; and collect the target data set from the target data provider through the target Source component.

[0130] In one embodiment of the present disclosure, the first acquisition module 403 is further configured to: identify, through the target Source component, that the data providing method of the data provider is to receive data sent by the data provider; and receive, through the target Source component, the target data set sent by the data provider.

[0131] In one embodiment of the present disclosure, the Flume system also includes a filter, and the device 400 also includes a processing module, which is configured to: filter the target data set through the target Source component to obtain a third data set; process the third data set from at least one target data provider through the filter to obtain the first data set.

[0132] In one embodiment of the present disclosure, the processing module is further configured to: obtain a data filtering code through the target Source component, wherein the data filtering code is a pre-set code for data filtering; execute the data filtering code through the target Source component, and perform data filtering on the target data set through the data filtering code to obtain a third data set.

[0133] In one embodiment of the present disclosure, the processing module is further configured to: obtain data filtering rules through the target Source component; filter out the first category of data that does not meet the data filtering rules from the target data set through the target Source component, and retain the second category of data that meets the data filtering rules to update the target data set; and use the updated target data set as the third data set through the target Source component.

[0134] In one embodiment of the present disclosure, the processing module is further configured to:

[0135] Sending target indication information to a target data provider of the first category of data source through the target Source component, wherein the target indication information is used to indicate that the first category of data does not meet the data screening rule, and / or to indicate that the first category of data is to be corrected;

[0136] Receiving, through the target Source component, first corrected data sent by a target data provider of the first type of data source, wherein the first corrected data is obtained by correcting the first type of data;

[0137] Identifying that the first revised data satisfies the data screening rule through the target Source component, and adding the first revised data to the third data set; or,

[0138] The target Source component identifies that the first revised data does not satisfy the data screening rule, updates the first revised data to the first category of data, and returns to execute the Source component to send the target indication information to the target data provider of the first category of data source and its subsequent steps.

[0139] In one embodiment of the present disclosure, the processing module is further configured to: perform correction processing on the first type of data according to the data screening rules through the target Source component to obtain second corrected data that meets the data screening rules; and add the second corrected data to the third data set through the target Source component.

[0140] In one embodiment of the present disclosure, the processing module is further configured to: obtain the user's data display requirements through the target Source component; and generate the data screening rules based on the data display requirements through the target Source component.

[0141] In one embodiment of the present disclosure, the device 400 also includes: a display module, configured to: obtain a fourth data set to be displayed by a target display device based on a second data set originating from at least one target data receiving end; and control the jumping or scrolling display of data in the fourth data set on the target display device.

[0142] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0143] In summary, the data processing device provided by the embodiment of the present disclosure can pre-develop multiple candidate Source components considering the categories of the candidate data providing ends, which are suitable for real-time data acquisition of various data providing ends, and pre-develop multiple candidate Sink components considering the categories of the candidate data receiving ends, which are suitable for real-time data storage of various data receiving ends. It has strong applicability, and determines the target Source component and the target Sink component therefrom. The target data set is obtained from the data providing end through the target Source component according to the data providing method of the target data providing end, and the second data set is stored in the data receiving end through the target Sink component according to the data storage method of the target data receiving end. The Flume system has the advantage of lightweight deployment, which ensures real-time data processing.

[0144] Figure 5 It is a block diagram of an electronic device according to an exemplary embodiment.

[0145] like Figure 5 As shown, the electronic device 500 includes:

[0146] The memory 510 and the processor 520, a bus 530 connecting different components (including the memory 510 and the processor 520), the memory 510 stores a computer program, and when the processor 520 executes the program, the data processing method described in the embodiment of the present disclosure is implemented.

[0147] Bus 530 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. For example, these architectures include but are not limited to Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus and Peripheral Component Interconnect (PCI) bus.

[0148] The electronic device 500 typically includes a variety of electronic device readable media. These media can be any available media that can be accessed by the electronic device 500, including volatile and non-volatile media, removable and non-removable media.

[0149] The memory 510 may also include computer system readable media in the form of volatile memory, such as random access memory (RAM) 540 and / or cache memory 550. The electronic device 500 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 560 may be used to read and write non-removable, non-volatile magnetic media ( Figure 5 not shown, usually called a "hard drive"). Although Figure 5Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 530 via one or more data medium interfaces. The memory 510 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present disclosure.

[0150] A program / utility 580 having a set (at least one) of program modules 570 may be stored, for example, in the memory 510, such program modules 570 including but not limited to an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules 570 generally perform the functions and / or methods of the embodiments described in the present disclosure.

[0151] The electronic device 500 may also communicate with one or more external devices 590 (e.g., keyboard, pointing device, display 591, etc.), one or more devices that enable a user to interact with the electronic device 500, and / or any device that enables the electronic device 500 to communicate with one or more other computing devices (e.g., network card, modem, etc.). Such communication may be performed through an input / output (I / O) interface 592. Furthermore, the electronic device 500 may also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN) and / or public network, such as the Internet) through a network adapter 593. Figure 5 As shown, the network adapter 593 communicates with other modules of the electronic device 500 via the bus 530. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the electronic device 500, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0152] The processor 520 executes various functional applications and data processing by running the programs stored in the memory 510 .

[0153] It should be noted that the implementation process and technical principles of the electronic device of this embodiment refer to the aforementioned explanation of the data processing method of the embodiment of the present disclosure, and will not be repeated here.

[0154] In summary, the electronic device provided by the embodiment of the present disclosure can execute the data processing method as described above, and can pre-develop multiple candidate Source components considering the categories of the candidate data providing ends, which are suitable for real-time data acquisition of various data providing ends, and pre-develop multiple candidate Sink components considering the categories of the candidate data receiving ends, which are suitable for real-time data storage of various data receiving ends, with strong applicability, and determine the target Source component and the target Sink component therefrom, and obtain the target data set from the data providing end through the target Source component according to the data providing method of the target data providing end, and store the second data set to the data receiving end through the target Sink component according to the data storage method of the target data receiving end, and the Flume system has the advantage of lightweight deployment, thereby ensuring real-time data processing.

[0155] In order to implement the above embodiments, the present disclosure also proposes a computer-readable storage medium.

[0156] When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device can perform the above-mentioned, or perform the above-mentioned data processing method. Optionally, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0157] In order to implement the above embodiments, the present disclosure further provides a computer program product, including a computer program, characterized in that when the computer program is executed by a processor, the data processing method as described above is implemented.

[0158] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0159] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A data processing method, characterized in that: The Flume system includes multiple candidate collector Source components, channel Channel components and multiple candidate receiver Sink components, wherein there is a corresponding relationship between the candidate Source components and the categories of the candidate data providers, there is a corresponding relationship between the candidate Sink components and the categories of the candidate data receivers, and the Channel components are in an activated state; The method comprises: Acquire a candidate Source component that corresponds to the category of the target data provider as a target Source component, and activate the target Source component; Acquire a candidate Sink component that has a corresponding relationship with the category of the target data receiving end as the target Sink component, and activate the target Sink component; Obtaining a target data set from the target data provider through the target Source component according to the data providing mode of the target data provider; storing, by the Channel component, a first data set to be stored, wherein the first data set is obtained by processing a target data set from at least one target data provider; Acquire a second data set to be stored in the target data receiving end from the first data set through the target Sink component, wherein the second data set is a subset of the first data set; The second data set is stored in the target data receiving end through the target Sink component according to the data storage mode of the target data receiving end.

2. The method according to claim 1, characterized in that The step of obtaining a target data set from the target data provider through the target Source component according to the data providing mode of the target data provider includes: Identifying, by the target Source component, that the data providing mode of the target data providing end is to collect data from the target data providing end; The target data set is collected from the target data provider through the target Source component.

3. The method according to claim 1, characterized in that The step of obtaining a target data set from the target data provider through the target Source component according to the data providing mode of the target data provider includes: Identifying, by the target Source component, that the data providing mode of the data providing end is to receive data sent by the data providing end; The target data set sent by the data provider is received through the target Source component.

4. The method according to claim 1, characterized in that: The Flume system further includes a filter, and the method further includes: Screening the target data set by using the target Source component to obtain a third data set; The filter is used to process a third data set from at least one target data provider to obtain the first data set.

5. The method according to claim 4, characterized in that The step of filtering the target data set by the target Source component to obtain a third data set includes: Obtaining a data screening code through the target Source component, wherein the data screening code is a pre-set code for data screening; The data screening code is executed through the target Source component, and the target data set is screened through the data screening code to obtain a third data set.

6. The method according to claim 4, characterized in that The step of filtering the target data set by the target Source component to obtain a third data set includes: Obtain data filtering rules through the target Source component; filtering out the first type of data that does not satisfy the data screening rule from the target data set through the target Source component, and retaining the second type of data that satisfies the data screening rule, so as to update the target data set; The updated target data set is used as the third data set through the target Source component.

7. The method according to claim 6, characterized in that The method further comprises: Sending target indication information to a target data provider of the first category of data source through the target Source component, wherein the target indication information is used to indicate that the first category of data does not meet the data screening rule, and / or to indicate that the first category of data is to be corrected; Receiving, through the target Source component, first corrected data sent by a target data provider of the first type of data source, wherein the first corrected data is obtained by correcting the first type of data; Identifying that the first revised data satisfies the data screening rule through the target Source component, and adding the first revised data to the third data set; or, The target Source component identifies that the first revised data does not satisfy the data screening rule, updates the first revised data to the first category of data, and returns to execute the Source component to send the target indication information to the target data provider of the first category of data source and its subsequent steps.

8. The method according to claim 6, characterized in that The method further comprises: The target Source component corrects the first type of data according to the data screening rule to obtain second corrected data that meets the data screening rule; The second revised data is added to the third data set through the target Source component.

9. The method according to any one of claims 6 to 8, characterized in that: The obtaining of data screening rules through the target Source component includes: Obtain the user's data display requirements through the target Source component; The data screening rules are generated by the target Source component based on the data display requirements.

10. The method according to any one of claims 1 to 8, characterized in that The method further comprises: Based on the second data set from at least one target data receiving terminal, obtaining a fourth data set to be displayed by the target display device; Control the jumping or scrolling display of the data in the fourth data set on the target display device.

11. A data processing device, characterized in that: The Flume system includes a plurality of candidate collector Source components, a channel Channel component and a plurality of candidate receiver Sink components, wherein there is a corresponding relationship between the candidate Source component and the category of the candidate data provider, there is a corresponding relationship between the candidate Sink component and the category of the candidate data receiver, the Channel component is in an activated state, and the device includes: A first activation module is configured to obtain a candidate Source component corresponding to the category of the target data provider as a target Source component, and activate the target Source component; The second activation module is configured to obtain a candidate Sink component having a corresponding relationship with the category of the target data receiving end as a target Sink component, and activate the target Sink component; A first acquisition module is configured to acquire a target data set from the target data provider through the target Source component according to the data providing mode of the target data provider; A first storage module is configured to store a first data set to be stored through the Channel component, wherein the first data set is obtained by processing a target data set from at least one target data provider; A second acquisition module is configured to acquire a second data set to be stored in the target data receiving end from the first data set through the target Sink component, wherein the second data set is a subset of the first data set; The second storage module is configured to store the second data set to the target data receiving end through the target Sink component according to the data storage method of the target data receiving end.

12. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the data processing method according to any one of claims 1 to 10.

13. A computer-readable storage medium, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the data processing method according to any one of claims 1 to 10.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the data processing method according to any one of claims 1 to 10 is implemented.