Data acquisition method and device, electronic equipment and storage medium

Abstract

The invention relates to a data acquisition method and device, electronic equipment and a storage medium. The data acquisition method comprises the steps that in response to an acquired data acquisition demand, a corresponding target theme is determined based on the data acquisition demand, and the target theme comprises target data corresponding to the data acquisition demand; a first node is called to correspondingly obtain the target data based on the target theme, the first node and the target theme have a corresponding relation, the first node is used for determining the target data based on first data, and the first data is data determined by a second node based on initial data; the second node is a producer of the first node. Through the method provided by the embodiment of the invention, the data acquisition process can be more stable.

Description

Technical Field

[0001] The present disclosure relates to the field of information processing, and in particular to a data acquisition method, device, electronic device and storage medium. Background Art

[0002] In big data scenarios, users usually obtain the required data based on platforms or tools (hereinafter referred to as "platforms" or "tools") that can process open source data streams.

[0003] However, in actual application, users may have various needs. For example, users may use the platform as a database to provide storage services for real-time data on the big data platform, or provide data sources for data analysis, or provide information that meets subscription needs for business parties. Summary of the invention

[0004] In order to overcome the problems existing in the related art, the present disclosure provides a data acquisition method, device, electronic device and storage medium.

[0005] According to a first aspect of an embodiment of the present disclosure, there is provided a data acquisition method, comprising: in response to obtaining a data acquisition requirement, determining a corresponding target topic based on the data acquisition requirement, wherein the target topic includes target data corresponding to the data acquisition requirement; calling a first node to obtain the target data corresponding to the target topic, wherein there is a corresponding relationship between the first node and the target topic, the first node is used to determine the target data based on first data, the first data is data determined by the second node based on initial data, and the second node is the producer of the first node.

[0006] In some embodiments, the initial data includes embedded point data and / or database data, and the first data is determined in the following manner: calling a third node to obtain the initial data, and setting the second node as a consumer of the third node; controlling the third node to send the initial data to the second node; in response to the initial data being embedded point data, calling the second node to deduplicate the embedded point data, and using the deduplicated data as the first data; in response to the initial data being database data, calling the second node to split the initial data, and using the split data as the first data.

[0007] In some embodiments, the initial data is buried point data, and the initial data corresponds to a sending priority, and the sending priority is the priority of the order in which the initial data is sent from the third node to the second node; before sending the initial data to the second node, the method also includes: monitoring the traffic value corresponding to the initial data; in response to the traffic value corresponding to the initial data being greater than or equal to a first traffic threshold, downgrading the initial data whose sending priority meets the priority requirement to obtain downgraded data, wherein the downgraded data is data that the third node does not send to the second node.

[0008] In some embodiments, the initial data is database data, and the initial data is written to the target storage area in the following manner: controlling the third node to determine the primary key identifier corresponding to the initial data based on the database identifier corresponding to the initial data; controlling the third node to store the initial data with the same primary key identifier in the same target storage area.

[0009] In some embodiments, the method further includes: monitoring the traffic value corresponding to the target topic; in response to the existence of at least two target topics whose traffic values ​​are greater than or equal to a second traffic threshold, setting at least two target topics to correspond to at least two first nodes.

[0010] According to a second aspect of an embodiment of the present disclosure, there is provided a data acquisition device, comprising: an acquisition unit, for determining a corresponding target subject based on a data acquisition requirement when a data acquisition requirement is obtained, wherein the target subject includes target data corresponding to the data acquisition requirement; a processing unit, for calling a first node to obtain the target data based on the target subject, wherein there is a corresponding relationship between the first node and the target subject, the first node is used to determine the target data based on first data, the first data is data determined by the second node based on initial data, and the second node is the producer of the first node.

[0011] In some embodiments, the initial data includes buried point data and / or database data, and the first data is determined in the following manner: calling a third node to obtain the initial data, and setting the second node as a consumer of the third node; controlling the third node to send the initial data to the second node; when the initial data is buried point data, calling the second node to deduplicate the buried point data, and using the deduplicated data as the first data; when the initial data is database data, calling the second node to split the initial data, and using the split data as the first data.

[0012] In some embodiments, the initial data is buried point data, and the initial data corresponds to a sending priority, and the sending priority is the priority of the order in which the initial data is sent from the third node to the second node; before sending the initial data to the second node, the device also includes: a first monitoring unit, used to monitor the traffic value corresponding to the initial data, and when the traffic value corresponding to the initial data is greater than or equal to a first traffic threshold, the device controls the processing unit to downgrade the initial data whose sending priority meets the priority requirement to obtain downgraded data, wherein the downgraded data is data that the third node does not send to the second node.

[0013] In some embodiments, the initial data is database data, and the initial data is written to the target storage area in the following manner: controlling the third node to determine the primary key identifier corresponding to the initial data based on the database identifier corresponding to the initial data; controlling the third node to store the initial data with the same primary key identifier in the same target storage area.

[0014] In some embodiments, the device also includes: a second monitoring unit, used to monitor the traffic value corresponding to the target topic, and in the case where there are at least two target topics whose traffic values ​​are greater than or equal to the second traffic threshold, the device controls the processing unit to set at least two target topics to correspond to at least two first nodes.

[0015] According to a third aspect of an embodiment of the present disclosure, a data acquisition device is provided, comprising: a processor; and a memory for storing processor executable instructions; wherein the processor is configured to: execute the data acquisition method described in the first aspect or any one of the implementations of the first aspect.

[0016] According to a fourth aspect of an embodiment of the present disclosure, a storage medium is provided, in which instructions are stored. When the instructions in the storage medium are executed by a processor, the processor is enabled to execute the data acquisition method described in the first aspect or any one of the embodiments of the first aspect.

[0017] The technical solution provided by the embodiments of the present disclosure may include the following beneficial effects: determining the target subject corresponding to the demand through the data acquisition demand, and then calling the first node corresponding to the target subject to obtain the target data corresponding to the target subject. Since there is a corresponding relationship between the target subject and the data acquisition demand, different data acquisition needs are met by the first nodes corresponding to different target subjects, thereby realizing the isolation and division of different business scenarios. Since the second node is the producer of the first node, the second node determines the first data based on the initial data, and the first node can determine the target data based on the first data. By setting nodes corresponding to different business processing contents and establishing a production-consumption relationship between the first node and the second node, and clarifying the division of responsibilities between the first node and the second node, a multi-level data acquisition structure is constructed to make the data acquisition process more stable.

[0018] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0020] Figure 1 The figure is a schematic diagram of a Kafka platform usage scenario according to an exemplary embodiment.

[0021] Figure 2A The figure is a flow chart of a data acquisition method according to an exemplary embodiment.

[0022] Figure 2B The figure is a flow chart of a data splitting method according to an exemplary embodiment.

[0023] Figure 2C The figure is a flow chart of a data acquisition method according to an exemplary embodiment.

[0024] Figure 3A The figure is a flowchart of a method for determining first data according to an exemplary embodiment.

[0025] Figure 3B The diagram is a scenario diagram showing a probe plug-in processing flow according to an exemplary embodiment.

[0026] Figure 3C It is a flow chart of a method for obtaining buried point data according to an exemplary embodiment.

[0027] Figure 3D The present invention is a flow chart showing a method of writing initial data into a target storage area according to an exemplary embodiment.

[0028] Figure 3E The figure is a schematic diagram of a database data reading and writing scenario according to an exemplary embodiment.

[0029] Figure 3F The diagram is a schematic diagram of a scenario showing a method for acquiring data according to an exemplary embodiment.

[0030] Figure 4 It is a schematic diagram of a scenario in which a user acquires data based on an operating platform, as proposed in an embodiment of the present disclosure.

[0031] Figure 5 The figure is a block diagram of a data acquisition device according to an exemplary embodiment.

[0032] Figure 6 It is a schematic diagram of the structure of an application embodiment of the electronic device disclosed in the present invention.

[0033] Figure 7 It is a schematic structural diagram of an embodiment of an electronic device disclosed in the present invention. DETAILED DESCRIPTION

[0034] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present disclosure.

[0035] In big data scenarios, users usually obtain the required data based on platforms or tools that can process open source data streams (hereinafter referred to as "platforms" or "tools"). However, in actual applications, users' needs may be diverse. For example, users use the platform as a database to provide storage services for real-time data on the big data platform. Or provide data sources for data analysis. Or provide information that meets subscription needs for business parties.

[0036] For ease of understanding, the following embodiments will use a distributed stream processing platform (eg, Kafka platform) as an example to schematically illustrate the usage scenarios of platforms or tools in big data scenarios by users in related technologies.

[0037] Figure 1 FIG. 1 is a schematic diagram of a Kafka platform usage scenario according to an exemplary embodiment. Figure 1As shown in the figure, in the big data scenario, users usually use the Kafka platform to process the following two types of business: 1) Acquire data. 2) Provide related subscription services based on the acquired data. For business 1), the acquired data types are divided by type, for example, they may include database data and / or buried point data. For business 2), the related subscription services provided include, for example: offline computing, real-time computing, log center, and business subscription.

[0038] However, through Figure 1 From the relevant examples in, we can see that, on the one hand, the scenarios in which users use the Kafka platform to process business are diverse, but there is no relevant isolation between the various business scenarios. This may lead to a decrease in the quality of platform services due to competition for the same resources for different business processing scenarios. On the other hand, the types of resources obtained by users based on the platform are diverse, but the process of obtaining resources is directly obtained through the Kafka platform. This may cause the platform to have greater pressure on data collection during the data collection process, and the data reusability is weak.

[0039] Based on this, in order to solve the above technical problems, the embodiment of the present disclosure proposes a data acquisition method. In response to the need for data acquisition, nodes matching the needs in the data acquisition platform are called based on different needs to achieve data acquisition, thereby realizing the isolation of the platform in multiple business scenarios. Among them, the nodes set in the data acquisition platform correspond to multiple levels, and the producer and consumer relationship between nodes at different levels is established, so that the platform can operate more stably during the data acquisition process.

[0040] Figure 2A is a flow chart of a data acquisition method according to an exemplary embodiment. Figure 2A As shown, the data acquisition method is used in a terminal and includes the following steps.

[0041] In step S11 , in response to obtaining the data acquisition requirement, a corresponding target subject is determined based on the data acquisition requirement, wherein the target subject includes target data corresponding to the data acquisition requirement.

[0042] In step S12, the first node is called to obtain corresponding target data based on the target topic, wherein there is a corresponding relationship between the first node and the target topic, the first node is used to determine the target data based on the first data, the first data is data determined by the second node based on the initial data, and the second node is the producer of the first node.

[0043] In the disclosed embodiment, the target topic (topic) corresponding to the demand is determined through the data acquisition demand, and then the first node corresponding to the target topic is called to obtain the target data corresponding to the target topic. Since there is a corresponding relationship between the target topic and the data acquisition demand, different data acquisition demands are met by the first nodes corresponding to different target topics, thereby realizing the isolation and division of different business scenarios. Since the second node is the producer of the first node, the second node determines the first data based on the initial data, and the first node can determine the target data based on the first data. By setting nodes corresponding to different business processing contents and establishing a production-consumption relationship between the first node and the second node, and clarifying the division of responsibilities between the first node and the second node, a multi-level data acquisition structure is constructed to make the data acquisition process more stable.

[0044] It can be understood that the first node and the second node respectively refer to internal nodes in the user's data acquisition platform (or tool).

[0045] For ease of understanding, the embodiments of the present disclosure will take the kafka platform as an example to illustrate the relevant embodiments of the present disclosure. It should be noted that the method proposed in the embodiments of the present disclosure is also applicable to other types of data acquisition platforms.

[0046] In some embodiments, the terms "node", "cluster", "physical cluster", "node set" and the like can be replaced with each other, for example, "first node" can be replaced with "first cluster", "second node" can be replaced with "second cluster", "third node" can be replaced with "third cluster", etc. It should be noted that the number of nodes mentioned in the embodiments of the present disclosure is not limited to "one node", and in some application scenarios, it can refer to "multiple nodes".

[0047] It should be noted that, for example, the setting of the first node can be based on the target subject in advance, and different first nodes can provide users with information that meets different needs. By establishing a correspondence between the first node and the target subject, in the actual application process, the target subject can be identified and the corresponding first node can be determined based on the set correspondence, so as to provide users with target data that meets data acquisition needs by calling the first node.

[0048] Exemplarily, the pre-set target topic is, for example, a target topic for supporting feature calculation. When the topic is determined by the user's data acquisition requirements, the corresponding first node will be called to obtain the target data corresponding to the target topic (for example, the target data for feature calculation).

[0049] In some embodiments, different data diversion rules may be set for different target topics so that different first nodes can acquire different target data.

[0050] Exemplarily, the second node produces corresponding target data for the first node based on different data diversion rules, wherein different data diversion rules correspond to different target topics.

[0051] To facilitate understanding, the following Figure 2B The process of the second node performing row data diversion based on the data diversion rule is described. Figure 2B FIG. 1 is a flow chart showing a data splitting method according to an exemplary embodiment. Figure 2B As shown, the second node may store the first data corresponding to multiple target topics (for example, db1, db2, and topic3, etc.). As the producer of the first node, the second node will distribute the data based on the preset diversion rules (for example, Figure 2B The different first data are diverted according to the diversion rules shown in the table to obtain the target data corresponding to the target subject, and based on the correspondence between the target subject and the first node, the target data is sent to the first node (the first node may store the acquired target data, or directly send the target data corresponding to the target subject to the user, etc.).

[0052] For the setting of data diversion rules, such as Figure 2B As shown in:

[0053] The data diversion rule with rule identifier 1: realizes the data merging of the data table field value of user in the topic of full subscription database (database, db) 1 and the data table field value of user in the topic of database 2, and writes it to the first node of the diversion customization corresponding to the target topic of user, thus realizing the data merging. Through the merge rule method, the data after database sharding can be synchronized to different Topics, and the aggregation of sharded data is realized through diversion, thus realizing the support for sharded scenarios.

[0054] The data diversion rule with rule identifier 2: The data table field value t2 of the subject full subscription database 1 is split separately and written to the first node of the corresponding target subject, thus realizing data splitting.

[0055] The data diversion rule with rule identifier 3: realizes the synchronization of all data subscribed to the Kafka cluster Topic topic3 ​​to the first diversion customized node Topic topic3_new, thus realizing full synchronization of data.

[0056] It is understandable that different target topics may correspond to different traffic sizes. Therefore, in some implementations, the corresponding relationship between the target topic and the first node may be established based on the traffic size of the target topic.

[0057] For example, Figure 2C FIG. 1 is a flow chart of a data acquisition method according to an exemplary embodiment. Figure 2C As shown, the following steps are included.

[0058] In step S21, the traffic value corresponding to the target topic is monitored.

[0059] In step S22, in response to the existence of at least two target topics whose traffic values ​​are greater than or equal to the second traffic threshold, the at least two target topics are set to correspond to at least two first nodes.

[0060] In the embodiment of the present disclosure, by monitoring the traffic value corresponding to the target topic, the traffic value of the target data corresponding to the target topic can be determined, and multiple target topics with traffic values ​​greater than the traffic threshold can be set as multiple first nodes respectively to reduce the situation where the data acquisition platform is unstable due to traffic tilt at the first node.

[0061] In summary, different first nodes can correspond to different target topics, so that different data acquisition requirements can be met by calling different first nodes.

[0062] The first data stored in the second node may be data determined by the second node based on the initial data. For example, the second node may determine the first data based on the initial data obtained by other nodes. It should be noted that the other nodes may be nodes that can obtain the initial data and can serve as producers of the second node, such as the third node.

[0063] Based on this, the initial data can be acquired by setting the third node, and the first data can be determined by the second node. Figure 3A A process of determining the first data is described. Figure 3A It is a flowchart of a method for determining first data according to an exemplary embodiment. The method includes the following steps.

[0064] In step S31, the third node is called to obtain initial data, and the second node is set as a consumer of the third node.

[0065] In step S32, the third node is controlled to send initial data to the second node.

[0066] In step S33-1, in response to the initial data being buried point data, the second node is called to perform deduplication processing on the buried point data, and the deduplication data is used as the first data.

[0067] In step S33 - 2 , in response to the initial data being database data, the second node is called to split the initial data, and the split data is used as the first data.

[0068] In the disclosed embodiment, the initial data is obtained by calling the third node, and the second node is called for corresponding processing according to different data types corresponding to the initial data, and the processed data is used as the first data. Due to the production-consumption relationship between the third node and the second node and the clear division of responsibilities between the third node and the second node, the third node can stably send the obtained initial data to the second node, and then the process of the second node determining the first data based on the obtained initial data can be stably executed.

[0069] In some embodiments, the initial data may be understood as an open source data stream that can be acquired by a third node, and its type may include, for example, embedded point data and / or database data. The third node may acquire the initial data based on a relevant data capture method. For example, the third node acquires the initial data based on a Change Data Capture (CDC) method.

[0070] In some embodiments, the acquisition of initial data, for example, the third node can be implemented by a probe plug-in. The probe plug-in can implement the core code as a variable, and / or the core data as a variable into the probe plug-in, and execute the user code block in the probe plug-in to implement the operations such as exception capture, data technology, and data collection in the probe plug-in.

[0071] In some embodiments, the probe plug-in can be embedded in the link segment of the data synchronization service for monitoring data collection. For example, the probe plug-in is embedded in the third node to obtain the link segment of the open source data flow for monitoring data collection. The probe plug-in can also be embedded in other data flow acquisition or data processing scenarios, such as being embedded in the second node to obtain the data of the third node and perform diversion processing.

[0072] For data acquisition based on the probe plug-in, for example, the following steps 1) to 3) may be included:

[0073] 1) Initialize the probe plugin.

[0074] 2) Control the probe plug-in for data acquisition.

[0075] 3) Based on the collected data type, control the probe plug-in to perform corresponding processing.

[0076] In some embodiments, for 1), for example, the parameters for acquiring data are set for the probe plug-in, and the parameters include, for example, a data source name, a system name, and a link name.

[0077] Exemplarily, the setting contents may refer to Table 1, for example.

[0078] Table 1

[0079] Parameter name Optional Example Data Source Name Required Mini Program Buried Data System Name Required Data Synchronization Link Name Required Data diversion

[0080] In some embodiments, for 2), for example, different types of probe plug-ins may be controlled to collect data. The probe plug-in types include at least one of the following:

[0081] Real-time counting probe: used to execute user code blocks, count the messages passed in by users, and then report the counts in batches or in real time to the probe data collection service.

[0082] Exception count and exception data detail probe: used to execute user code blocks, capture user code block execution exceptions, count the messages where exceptions occur, and then report the counts in batches or in real time to the probe data service; at the same time, report the content of the message where the exception occurs to the probe data collection service.

[0083] Data delay counting probe: used to obtain the data time of the message and calculate the difference between the current system time based on the time data field of the message, and report the difference and the current system time to the probe data collection service.

[0084] Data direct reporting probe: used to directly report other types of reporting data for data indicators, such as data deduplication and data degradation message counts, directly reporting to the probe data collection service.

[0085] In some embodiments, the input parameters provided by the above different types of probe plug-ins may be as shown in Table 2, for example:

[0086] Table 2

[0087]

[0088] Among them, "XX" represents the parameters in the information or code block, and "{}" represents the optional input of the parameter.

[0089] It is understandable that different types of probe plug-ins may obtain different types of data. Based on this, it is necessary to set corresponding processing flows for the data obtained by different types of probe plug-ins.

[0090] Exemplarily, for data acquired by a data delay counting probe, the acquired point-by-point data may be written into a time series database.

[0091] Exemplarily, the data acquired by the real-time counting probe will be directly reported to the corresponding analysis database.

[0092] Exemplarily, for the data acquired by the abnormal count and abnormal data detail probes, the data corresponding to the abnormal count will be written into the analysis database; and the reported abnormal message will be written into the document database for abnormal message troubleshooting.

[0093] For ease of understanding, the following will be Figure 3B The processing flow corresponding to the probe plug-in is schematically described. Figure 3B FIG. 1 is a schematic diagram of a probe plug-in processing flow according to an exemplary embodiment. Figure 3B As shown, take the real-time counting probe (i.e., the real-time probe in the figure) as an example. The real-time probe will obtain the data generated by the real-time program, such as the core data processing code block and real-time message input shown in the figure. After obtaining the code block and data information, the real-time probe will monitor and capture abnormal data, and report the obtained data to the relevant service (such as the unified probe collection service in the figure), and then write the data to the analysis database through the service. As for the reporting method, for example, batch reporting and / or real-time reporting can be used.

[0094] After acquiring the data, you can also perform data quantity analysis based on the acquired data, for example, generate a quantity data report based on the data acquired by the real-time counting probe, determine the data delay based on the data acquired by the delayed counting probe, and perform data anomaly analysis based on the data acquired by the abnormal count and abnormal data detail probes.

[0095] Exemplarily, for data obtained based on real-time counting probes, a quantity data report is generated. For example, the data source and flow path written to the corresponding link end can be obtained through the data processing platform, and combined with the probe plug-in collection technology data of each link segment, the total number of messages in any link segment within the specified time range and the corresponding abnormal information can be determined, and finally a quantity data report is generated based on the full link data. The quantity data report may include the following content: select any multi-end links and compare the data indicators of the multi-end data. Such as the data volume of the A end, the data volume of the B end, the upstream and downstream (AB) difference comparison, the upstream and downstream (AB) difference rate, the (AB) end degraded data, and the (AB) end deduplication data.

[0096] Exemplarily, the data delay situation is determined based on the data obtained by the delay counting probe. For example, the delay dot data in the time series database can be used to draw a delay curve for the link segment time range, and the data delay situation of the current link segment is determined based on the delay curve.

[0097] Exemplarily, data anomaly analysis is performed on data acquired based on anomaly counts and abnormal data detail probes. For example, data fluctuations in different time ranges may be calculated based on the acquired data on a year-on-year and month-on-month basis.

[0098] In some embodiments, for data acquired based on the probe plug-in, a corresponding threshold may be set for the link acquired by the probe plug-in for monitoring. For example, the set threshold is a data volume threshold, and when the data volume of the corresponding link acquired by the probe plug-in is greater than the data volume threshold, a relevant prompt message is sent to the user.

[0099] Based on this, a full-link data matching and monitoring method is implemented, and the data collected by the probe data provides a data basis for the full-link data matching and monitoring analysis. Combined with the data analysis engine and method, the link problem points are quickly identified from the overall link, thereby improving the link stability. It is understandable that some duplicate data or useless data may exist in the initial data captured by the third node. In order to enable the data processing platform to operate stably, reduce the resources occupied by some duplicate data or useless data in the initial data, and to increase the reusability of the data. The initial data obtained by the first node can be processed accordingly by the second node to obtain the first data.

[0100] For example, in some embodiments, different row data processing actions may be performed on different types of initial data by calling the second node.

[0101] For example, for the buried point data in the initial data, the second node can be called to perform data deduplication processing on it, and the deduplication data can be used as the first data. For the data in the database, the second node can be called to split it (for example, classification, merging, etc.) and the split data can be used as the first data.

[0102] It is understandable that the buried data includes, for example, client logs or server logs. It is usually triggered by user behavior, with a large amount of data, high real-time requirements, and there may be duplicate reports. If the buried data is directly synchronized to multiple consumers, it may cause resource waste and performance bottlenecks. Therefore, for the synchronization needs of buried data, it is usually necessary to ensure the uniqueness of the data to avoid duplicate data affecting the accuracy of the analysis results. That is, deduplication is a key requirement for the synchronization of buried data, which is usually achieved through a unique identifier of the message (such as event-guid).

[0103] For example, when different terminals report data, a unique identifier event-guid of the message will be generated, and the second node will deduplicate the data based on the event-guid. For example, the state implementation based on the open source stream processing framework Flink supports deduplication of data on the same day. During the message processing process, it is determined whether the message exists in the state. If it does, the data is deleted. If it does not exist, the event-guid is written into the state. The cleanup of the state on the same day will be triggered regularly on the next day.

[0104] It is known that the data flow of buried data is usually large, and the data with large data flow of buried data may not be critical data. Therefore, in order to prevent the third node from obtaining a large amount of non-critical buried data, the priority of obtaining some non-critical data can be reduced when the buried data flow is greater than a certain flow threshold.

[0105] For example, Figure 3C FIG. 1 is a flow chart of a method for obtaining buried point data according to an exemplary embodiment. Figure 3C As shown, the following steps are included.

[0106] In step S41, the flow value corresponding to the initial data is monitored.

[0107] In step S42, in response to the traffic value corresponding to the initial data being greater than or equal to the first traffic threshold, the initial data whose sending priority meets the priority requirement is downgraded to obtain downgraded data, wherein the downgraded data is data that the third node does not send to the second node.

[0108] Among them, the initial data is the buried point data, the initial data corresponds to the sending priority, and the sending priority is the priority of the initial data in the order in which it is sent from the third node to the second node.

[0109] In the embodiment of the present disclosure, by performing flow detection on the initial data, and when the flow value of the initial data is greater than or equal to the first flow threshold, the initial data that meets the transmission priority requirement is downgraded so as not to send the data to the second node. This can reduce the non-important initial data with large flow obtained by the second node, thereby ensuring the stability of the data acquisition process.

[0110] It should be noted that for the sending priority to satisfy the priority requirement, for example, the sending priority may be lower than a certain priority threshold.

[0111] It is also understandable that database data is usually structured business data, such as user information, order information, etc., with high data accuracy requirements and relatively orderly changes (for example, updating or deleting records by primary key). Therefore, for the synchronization requirements of database data, it is usually necessary to ensure the consistency and order of data, especially in the scenario of distributed database or sharding database and table, data synchronization needs to maintain the integrity and order of data.

[0112] In order to ensure the data integrity and order during the database data synchronization process, for example, the storage structure design can be implemented by calling the third node to store the database data. Figure 3D As shown, Figure 3D The present invention is a flow chart of a method for writing initial data into a target storage area according to an exemplary embodiment, which includes the following steps.

[0113] In step S51, the third node is controlled to determine a primary key identifier corresponding to the initial data based on a database identifier corresponding to the initial data.

[0114] In step S52, the third node is controlled to store the initial data with the same primary key identifier into the same target storage area.

[0115] In the embodiment of the present disclosure, the third node is controlled to determine the primary key identifier corresponding to the initial data based on the database identifier corresponding to the initial data, and then the corresponding initial data is stored in the target storage area according to the primary key identifier, wherein the initial data with the same primary key identifier is stored in the same target storage area.

[0116] In some embodiments, for the primary key identifier, for example, the value of the database name (schema) in the database data field can be concatenated with the table value, and then concatenated with each value of the primary key list in the primary key field in the field.

[0117] It can be understood that after obtaining the primary key identifier, the primary key identifier is hashed, and the calculated result is modulo the number corresponding to the target storage area (or the number of partitions corresponding to the same target topic), and the final calculated value is used as the number of the message written into the target storage area. This method can write data with the same unique primary key identifier into the same target storage area, that is, data with the same primary key value in the same library table will be written to the same target storage area.

[0118] For ease of understanding, we will Figure 3E The reading and writing process of database data is illustrated by way of example. Figure 3E FIG. 1 is a schematic diagram of a database data reading and writing scenario according to an exemplary embodiment. Figure 3EAs shown, assuming that the database name is "user", the table name is "userinfo", the identifier (id) is used as the primary key, and the number of target storage areas is 3, the database data generated with id 1 is used as an example:

[0119] Message 2 (the message whose before-value corresponds to 100 and after-value corresponds to 110) has a unique message identifier user-userinfo=1, which is obtained by hash operation modulo hash(user-userinfo-1)%3=1. The message is written into the target storage area numbered 1.

[0120] Message 4 (the message whose before-value corresponds to 110 and after-value corresponds to 120) has a unique message identifier of user-userinfo=1, which is obtained by hash operation modulo hash(user-userinfo-1)%3=1. The message is written into the target storage area numbered 1.

[0121] Finally, the two messages with id 1 are written into the target storage area numbered 1 in sequence. For the consumer program, since the same message is in the same partition, the order of reading can be guaranteed to be consistent with the order of writing. So far, the writing and writing process is orderly as a whole.

[0122] Based on this, the first data stored in the second node can be determined through the association between the second node and the third node.

[0123] It is understood that in some embodiments, Figure 2A to Figure 2C as well as Figure 3A The embodiments corresponding to Figures 1 to 4 can be implemented in combination with each other. Figure 3F The application scenarios of the optional implementation modes obtained by combining the above embodiments are schematically described. Figure 3F The diagram is a schematic diagram of a scenario showing a method for acquiring data according to an exemplary embodiment.

[0124] like Figure 3FAs shown, the buried data and database data can be obtained through the third node (for example, database data is obtained based on CDC). The third node, as the producer of the second node, sends the acquired buried data and database data to the second node. The second node can process the acquired original data accordingly to obtain the first data, and store the obtained first data. As the producer of the first node, the second node can send the first data to the first node of the corresponding target topic based on the preset diversion rules. In this way, when the data acquisition requirements are obtained, the target topic that matches the data acquisition requirements can be determined, and then the corresponding first node is determined according to the determined target topic, so as to obtain the corresponding target data.

[0125] It is understandable that Figure 3F The description of the relevant embodiments has been introduced relatively similarly in the aforementioned embodiments. For related contents, please refer to the aforementioned corresponding embodiment description, which will not be repeated here one by one.

[0126] Based on this, the above embodiments can realize stable acquisition of target data by users through a multi-level data acquisition platform, thereby improving the user's experience of using the data acquisition platform.

[0127] In some embodiments, for the above data acquisition method, for example, a user can implement it through a related operating platform. Figure 4 It is a schematic diagram of a scenario in which a user acquires data based on an operating platform, as proposed in an embodiment of the present disclosure.

[0128] exist Figure 4 In the process, users can raise relevant data acquisition requirements through the operation platform, such as Figure 4 In the data subscription application, the subject management application and the monitoring subscription application, etc. The operation platform determines the corresponding target subject according to the user's needs. For example, the target subject corresponding to the data subscription application is "subscription information", the target subject corresponding to the subject management application is "subscription information", and the target subject corresponding to the monitoring subscription application is "subscription information".

[0129] After the operating system obtains the target topic corresponding to the user's data acquisition needs, it sends an approval request to the administrator in the form of a platform work order, and creates account information that can be used to obtain data (such as creating a tenant account in the figure) to perform corresponding data acquisition operations if the approval is passed. In some embodiments, different operating permissions may correspond to different tenant accounts created, so it is necessary to authorize resources and operations for the account. For example, user A wants to use consumer group B to consume data with a target topic of "topic operation information." For this scenario, the permission management process is as follows: Dynamically create users: Create a corresponding operating user and assign a password. The account is authorized to consume data with a target topic of "topic operation information", but is not authorized to consume data with a target topic of "topic subscription information."

[0130] In some embodiments, the operating system may also be used to monitor information and push reports to users.

[0131] In the disclosed embodiment, the present invention first divides the multi-level nodes according to the functional and scenario dimensions for physical and scenario isolation, and at the same time, the target subject is reasonably divided into different nodes according to the actual load situation in the cluster, so as to realize the resource isolation within the node, and at the same time, avoid the problem of tilt caused by uneven traffic on different nodes, and improve the reliability of the data acquisition system service. Further, the data synchronization function design realizes the ability of data synchronization, data deduplication, data splitting and merging, greatly improves the data reusability and data processing efficiency of database data and buried points, and reduces the database pressure. For database data, data merging realizes the support for the sub-library and sub-table scenarios, and at the same time, the data synchronization process realizes the order based on the primary key granularity, which also meets the support for data order and more scenarios. For buried point data, it realizes the availability of core data through one-key downgrade in the data synchronization process. In summary, the application design of the data acquisition system solves the common use problems of database data and buried point data in big data scenarios, such as improving the overall availability of the link, improving data reusability and data usage efficiency, and reducing database pressure in reading and consumption, and the overall application design has industry universality.

[0132] Furthermore, by standardizing the data processing and usage processes and implementing the data acquisition platform in a systematic way, many links such as account management, user resource operation application, work order approval, platform user authorization, monitoring and report push have been automated, achieving unified platform management and control, solving core issues such as standardized use, platform unified management and control processes, process automation, and data security, greatly improving the platform's control capabilities and user efficiency.

[0133] Based on the same concept, an embodiment of the present disclosure also provides a data acquisition device.

[0134] It is understandable that the data acquisition device provided by the embodiment of the present disclosure includes hardware structures and / or software modules corresponding to the execution of each function in order to realize the above functions. In combination with the units and algorithm steps of each example disclosed in the embodiment of the present disclosure, the embodiment of the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the technical solution of the embodiment of the present disclosure.

[0135] Figure 5 is a block diagram of a data acquisition device according to an exemplary embodiment. Figure 5 The device 100 includes an acquisition unit 101 and a processing unit 102.

[0136] The acquisition unit 101 is used to determine a corresponding target topic based on the data acquisition requirement when a data acquisition requirement is acquired, wherein the target topic includes target data corresponding to the data acquisition requirement.

[0137] The processing unit 102 is used to call the first node to obtain target data based on the target topic, wherein there is a corresponding relationship between the first node and the target topic, the first node is used to determine the target data based on the first data, the first data is the data determined by the second node based on the initial data, and the second node is the producer of the first node.

[0138] In some embodiments, the initial data includes buried point data and / or database data, and the first data is determined in the following manner: calling a third node to obtain the initial data, and setting the second node as the consumer of the third node; controlling the third node to send the initial data to the second node; when the initial data is buried point data, calling the second node to deduplicate the buried point data, and using the deduplicated data as the first data; when the initial data is database data, calling the second node to split the initial data, and using the split data as the first data.

[0139] In some embodiments, the initial data is buried point data, and the initial data corresponds to a sending priority, and the sending priority is the priority of the order in which the initial data is sent from the third node to the second node; before sending the initial data to the second node, the device also includes: a first monitoring unit, used to monitor the traffic value corresponding to the initial data, and when the traffic value corresponding to the initial data is greater than or equal to the first traffic threshold, the device controls the processing unit to downgrade the initial data whose sending priority meets the priority requirement to obtain downgraded data, wherein the downgraded data is data that the third node does not send to the second node.

[0140] In some embodiments, the initial data is database data, and the initial data is written to the target storage area in the following manner: the third node is controlled to determine the primary key identifier corresponding to the initial data based on the database identifier corresponding to the initial data; the third node is controlled to store the initial data with the same primary key identifier in the same target storage area.

[0141] In some embodiments, the device also includes: a second monitoring unit, used to monitor the traffic value corresponding to the target topic, and in the case where there are at least two target topics whose traffic values ​​are greater than or equal to the second traffic threshold, the device controls the processing unit to set the at least two target topics to correspond to at least two first nodes.

[0142] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0143] Figure 6 FIG. 1 is a schematic diagram of a structure of an application embodiment of the electronic device disclosed in the present invention. Figure 6 The architecture shown is only an optional implementation method. In the specific practice, the above can be modified according to actual needs. Figure 6 The number and type of components can be selected, deleted, added or replaced; different functional components can be set up by separate settings or integrated settings, for example, the GPU and CPU can be set up separately or the GPU can be integrated on the CPU, the communication unit can be set up separately or integrated on the CPU or GPU, etc. These replaceable implementations all fall within the protection scope disclosed in the present invention.

[0144] Figure 7 Schematic diagram of the structure of an embodiment of the electronic device disclosed in the present invention. Figure 7 , which shows a schematic diagram of the structure of an electronic device suitable for implementing a terminal device or a server in an embodiment of the present application. Figure 7 As shown, the electronic device includes a processor and a memory. The electronic device may also include an input and output device. The memory and the input and output device are connected to the processor through a bus. Among them, the memory is used to store instructions executed by the processor; the processor is used to call the instructions stored in the memory and execute the method of associating tag content involved in the above embodiment.

[0145] In the disclosed embodiment, the processor may call the instructions stored in the memory to determine the tag to be associated; determine the circled content in the data content according to the semantic matching result between the tag to be associated and the word tag marked by the data content; and associate the circled content with the tag to be associated. The process of the electronic device executing the tag-associated content can refer to the implementation process of the tag-associated content method described in the above embodiment, which will not be repeated here.

[0146] The embodiment of the present disclosure further provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are run on a computer, the method of associating tags with content involved in the above embodiment is executed.

[0147] The embodiments of the present disclosure also provide a computer program product including instructions. When the computer program product including instructions is run on a computer, the computer is enabled to execute the method of associating tags with content involved in the above embodiments.

[0148] In one or more optional implementations, the disclosed embodiment further provides a computer-readable storage medium for storing computer-readable instructions, which, when executed, causes a computer to execute the method of associating tag content in any of the above possible implementations. In another optional example, the computer program product is specifically embodied as a software product, such as a software development kit (SDK) and the like.

[0149] Although operations are described in a particular order in the drawings, this should not be understood as requiring that the operations be performed in the particular order shown or in serial order, or that all shown operations be performed to achieve desired results. In certain circumstances, multitasking and parallel processing may be advantageous.

[0150] The methods and apparatus of the present disclosure can be implemented using standard programming techniques, using rule-based logic or other logic to implement various method steps. It should also be noted that the words "apparatus" and "module" used herein and in the claims are intended to include implementations using one or more lines of software code and / or hardware implementations and / or devices for receiving input.

[0151] Any steps, operations or procedures described herein may be performed or implemented using one or more hardware or software modules, either alone or in combination with other devices. In one embodiment, the software module is implemented using a computer program product including a computer-readable medium containing computer program code, which can be executed by a computer processor to perform any or all of the described steps, operations or procedures.

[0152] The foregoing description of the implementation of the present disclosure has been given for the purpose of illustration and description. The foregoing description is not exhaustive nor is it intended to limit the present disclosure to the exact form disclosed, and various variations and modifications may exist in accordance with the above teachings or may be obtained from the practice of the present disclosure. These embodiments are selected and described in order to illustrate the principles of the present disclosure and its practical application, so that those skilled in the art can utilize the present disclosure in various embodiments and various modifications suitable for the specific purpose contemplated.

[0153] It is to be understood that in the present disclosure, "plurality" refers to two or more than two, and other quantifiers are similar. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. The singular forms "a", "the" and "the" are also intended to include plural forms, unless the context clearly indicates other meanings.

[0154] It is further understood that the terms "first", "second", etc. are used to describe various information, but such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other, and do not indicate a specific order or degree of importance. In fact, the expressions "first", "second", etc. can be used interchangeably. For example, without departing from the scope of the present disclosure, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information.

[0155] It can be further understood that, unless otherwise specified, “connection” includes a direct connection without other components between the two, and also includes an indirect connection with other components between the two.

[0156] It is further understood that, although the operations are described in a specific order in the drawings in the embodiments of the present disclosure, it should not be understood as requiring the operations to be performed in the specific order shown or in a serial order, or requiring the execution of all the operations shown to obtain the desired results. In certain environments, multitasking and parallel processing may be advantageous.

[0157] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any modifications, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure.

[0158] It should be understood that the present disclosure is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the scope of the appended claims.

Claims

1. A data acquisition method, characterized in that: include: In response to obtaining the data acquisition requirement, determining a corresponding target topic based on the data acquisition requirement, wherein the target topic includes target data corresponding to the data acquisition requirement; Call the first node to obtain the target data based on the target topic, wherein there is a corresponding relationship between the first node and the target topic, the first node is used to determine the target data based on first data, the first data is data determined by the second node based on initial data, and the second node is the producer of the first node.

2. The method according to claim 1, characterized in that: The initial data includes buried point data and / or database data, and the first data is determined in the following manner: Calling a third node to obtain the initial data, and setting the second node as a consumer of the third node; Controlling the third node to send the initial data to the second node; In response to the initial data being buried point data, calling the second node to perform deduplication processing on the buried point data, and using the deduplication data as the first data; In response to the initial data being database data, the second node is called to split the initial data and use the split data as the first data.

3. The method according to claim 2, characterized in that The initial data is buried point data, the initial data corresponds to a sending priority, and the sending priority is the priority of the initial data in the order in which it is sent from the third node to the second node; Before sending the initial data to the second node, the method further includes: Monitoring the flow value corresponding to the initial data; In response to the traffic value corresponding to the initial data being greater than or equal to a first traffic threshold, the initial data whose sending priority meets the priority requirement is downgraded to obtain downgraded data, wherein the downgraded data is data that the third node does not send to the second node.

4. The method according to claim 2, characterized in that: The initial data is database data, The initial data is written to the target storage area in the following manner: Control the third node to determine a primary key identifier corresponding to the initial data based on a database identifier corresponding to the initial data; The third node is controlled to store the initial data with the same primary key identifier in the same target storage area.

5. The method according to claim 1, characterized in that The method further comprises: Monitoring the traffic value corresponding to the target topic; In response to the existence of at least two target topics whose traffic values ​​are greater than or equal to the second traffic threshold, the at least two target topics are set to correspond to the at least two first nodes.

6. A data acquisition device, characterized in that: include: An acquisition unit, configured to determine a corresponding target subject based on the data acquisition requirement when a data acquisition requirement is acquired, wherein the target subject includes target data corresponding to the data acquisition requirement; A processing unit is used to call the first node to obtain the target data based on the target topic, wherein there is a corresponding relationship between the first node and the target topic, the first node is used to determine the target data based on first data, the first data is data determined by the second node based on initial data, and the second node is the producer of the first node.

7. The device according to claim 6, characterized in that The initial data includes buried point data and / or database data, and the first data is determined in the following manner: Calling a third node to obtain the initial data, and setting the second node as a consumer of the third node; Controlling the third node to send the initial data to the second node; In the case where the initial data is buried point data, calling the second node to perform deduplication processing on the buried point data, and using the deduplication data as the first data; When the initial data is database data, the second node is called to split the initial data and the split data is used as the first data.

8. The device according to claim 7, characterized in that The initial data is buried point data, the initial data corresponds to a sending priority, and the sending priority is the priority of the initial data in the order in which it is sent from the third node to the second node; Before sending the initial data to the second node, the device further includes: The first monitoring unit is used to monitor the traffic value corresponding to the initial data, and when the traffic value corresponding to the initial data is greater than or equal to the first traffic threshold, the device controls the processing unit to downgrade the initial data whose sending priority meets the priority requirement to obtain downgraded data, wherein the downgraded data is data that the third node does not send to the second node.

9. The device according to claim 7, characterized in that The initial data is database data, The initial data is written to the target storage area in the following manner: Control the third node to determine a primary key identifier corresponding to the initial data based on a database identifier corresponding to the initial data; The third node is controlled to store the initial data with the same primary key identifier in the same target storage area.

10. The device according to claim 6, characterized in that The device also includes: The second monitoring unit is used to monitor the traffic value corresponding to the target topic, and when there are at least two target topics whose traffic values ​​are greater than or equal to the second traffic threshold, the device controls the processing unit to set the at least two target topics to correspond to at least two first nodes.

11. An electronic device, characterized in that: include: processor: a memory for storing processor-executable instructions; Wherein, the processor is configured to: execute the data acquisition method according to any one of claims 1 to 5.

12. A storage medium, characterized in that: The storage medium stores instructions. When the instructions in the storage medium are executed by a processor, the processor is enabled to execute the data acquisition method according to any one of claims 1 to 5.

Images