Method and device for delayed association of streaming data and dimensional data
By creating connection pool and dimension data cache objects, listening and retrying the association between streaming data and dimension data, the data statistics error problem caused by failure of streaming data and dimension data is solved, and the adaptation of various data sources is achieved. Savings in middleware overhead.
Patent Information
- Application Number
- CN202411365371.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-29
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-09-29
AI Technical Summary
In big data scenarios, the failure of the association between streaming data and dimensional data leads to incorrect data statistics results. The existing solutions require special configuration of the data source and the introduction of additional message delay middleware to increase middleware overhead.
By creating the connection pool and dimension data cache object of the first dimension table database, listen to and read the streaming data to be associated. If the association fails, enter the waiting retry mode until the retry waiting time reaches the latest time, re-execute the association operation again. Ensure that streaming data and dimensional data are successfully associated.
There is no need to make special configurations of the data source or introduce additional middleware to effectively solve the problem of data statistics errors caused by failed association of streaming data and dimensional data, saving middleware overhead.
Smart Images

Figure CN118885514B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing, and in particular to a method and device for delay association of streaming data and dimensional data. Background Art
[0002] In big data scenarios, in order to improve the efficiency of data query, streaming data is usually associated with necessary dimensional data in advance before storage when it arrives. In this way, when querying data later, you can directly query based on the associated results, thus saving the time and computing resources of data association. At the same time, for high-frequency queries, the cost of performing an association before writing is much less than the cost of temporary association at each query.
[0003] If the streaming data reaches the data side, but the data side fails to synchronize the dimensional data corresponding to the streaming data from the business side in time, the dimensional data corresponding to the streaming data will be empty due to association failure. However, when counting data, statistics are usually based on some dimensional information (dimensional data) of the task, such as task name, task creation time, etc. If the streaming data and dimensional data fail to associate as described above, then the streaming data that cannot be associated with the task dimensional information (dimensional data) (such as the subtasks of OTA (Over the Air Technology) upgrade) will be missed, resulting in errors in the statistical results.
[0004] In response to the above technical problems, the traditional solution is mainly to use the built-in delay queue function provided by some middleware (such as Apache Pulsar, RabbitMQ delay plug-in, etc.) to achieve delayed consumption of messages. However, for data sources that do not support delay queues, this solution requires special modifications to the data source configuration, etc., and requires the introduction of additional message delay middleware, thereby increasing additional middleware overhead.
[0005] Therefore, there is an urgent need to provide an association solution that can adapt to various data sources without making special modifications to the data source configuration, or introducing additional message delay middleware, thereby saving additional middleware overhead, and can effectively solve the problem of subsequent data statistical results being erroneous due to the failure to associate streaming data with dimensional data. Summary of the invention
[0006] In view of this, the present application provides a method and device for delayed association of streaming data and dimensional data, so as to achieve adaptability to various data sources without making special modifications to the configuration of the data source, and without introducing additional message delay middleware, thereby saving additional middleware overhead and effectively solving the problem of errors in subsequent data statistical results caused by failure to associate streaming data with dimensional data.
[0007] In a first aspect of the present application, a delayed association method for streaming data and dimensional data is provided, comprising:
[0008] Create a connection pool and dimension data cache object corresponding to the first dimension table database, and synchronize the first dimension data of the first dimension table database with the second dimension data of the second dimension table database on the business end;
[0009] Listening to and reading the current streaming data to be associated sent by the first terminal, and recording the data reading timestamp and data type identifier of the current streaming data to be associated;
[0010] If it is determined that the current streaming data to be associated fails to be associated with the dimension data cache object and the first dimension table database, and the current streaming data to be associated meets the preset waiting and retrying conditions, then the retry waiting time after the current streaming data to be associated enters the waiting and retrying mode is determined;
[0011] If the retry waiting time reaches the current latest waiting time, the association operation between the current streaming data to be associated and the first dimension table database is re-executed;
[0012] If the current streaming data to be associated is successfully associated with the first dimension table database, the first target dimension data associated with the current streaming data to be associated is added to the dimension data cache object, and the target streaming data is saved and output. The target streaming data consists of the current streaming data to be associated and its associated first target dimension data.
[0013] In a second aspect of the present application, a device for delaying association of streaming data and dimensional data is provided, comprising:
[0014] A creation module is configured to create a connection pool and a dimension data cache object corresponding to the first dimension table database, and synchronize the first dimension data of the first dimension table database with the second dimension data of the second dimension table database of the business end;
[0015] A reading module is configured to monitor and read the current streaming data to be associated sent by the first terminal, and record a data reading timestamp and a data type identifier of the current streaming data to be associated;
[0016] The waiting module is configured to determine the retry waiting time after the current streaming data to be associated enters the waiting retry mode if it is determined that the current streaming data to be associated fails to be associated with the dimension data cache object and the first dimension table database, and the current streaming data to be associated meets the preset waiting retry condition;
[0017] A retry module is configured to re-execute the association operation between the current streaming data to be associated and the first dimension table database if the retry waiting time reaches the current latest waiting time;
[0018] The output module is configured to add the first target dimension data associated with the current streaming data to be associated to the dimension data cache object if the current streaming data to be associated is successfully associated with the first dimension table database, save and output the target streaming data, and the target streaming data consists of the current streaming data to be associated and its associated first target dimension data.
[0019] According to a third aspect of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.
[0020] According to a fourth aspect of the present application, a readable storage medium is provided, which stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.
[0021] Compared with the prior art, the beneficial effects of the present application include at least: by creating a connection pool and a dimension data cache object corresponding to the first dimension table database on the server side; then, when it is determined that the association between the current streaming data to be associated and the dimension data cache object and the first dimension table database fails, but the preset waiting and retry conditions are met, determining the retry waiting time after the current streaming data to be associated enters the waiting and retry mode; when the retry waiting time reaches the current latest waiting time, re-executing the association operation between the current streaming data to be associated and the first dimension table database, thereby effectively solving the problem of errors in subsequent data statistical results caused by the failure to associate streaming data with dimension data, and the solution can adapt to various data sources without making special modifications to the configuration of the data source, etc., and without introducing additional message delay middleware, which can save additional middleware overhead. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0023] Figure 1 It is a schematic diagram of an application scenario of an embodiment of the present application;
[0024] Figure 2 It is a schematic diagram of the automobile OTA upgrade strategy and task creation and issuance process provided in the embodiment of the present application;
[0025] Figure 3 It is a flow chart of associating streaming data and dimensional data using a streaming processing method provided in an embodiment of the present application;
[0026] Figure 4 It is a flowchart of a delayed association method of streaming data and dimensional data provided by an embodiment of the present application;
[0027] Figure 5 It is a flowchart of determining whether the current streaming data to be associated meets the preset waiting retry condition provided by an embodiment of the present application;
[0028] Figure 6 This is a flow chart of determining the current latest waiting time provided by an embodiment of the present application;
[0029] Figure 7 It is a comparative schematic diagram of the data processing flow of a strict mode and a normal mode provided in an embodiment of the present application;
[0030] Figure 8 It is a flowchart of another delayed association method of streaming data and dimensional data provided in an embodiment of the present application;
[0031] Fig. 9 It is a structural schematic diagram of a delayed association device for streaming data and dimensional data provided in an embodiment of the present application;
[0032] Fig.10 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0033] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.
[0034] A method and device for delay association of streaming data and dimensional data according to an embodiment of the present application will be described in detail below with reference to the accompanying drawings.
[0035] Figure 11 is a schematic diagram of an application scenario of an embodiment of the present application. The application scenario may include a server 101, a service end 102, a first terminal 103, and a second terminal 104. The server 101 and the service end 102, the first terminal 103, and the second terminal 104 may be connected via a network communication; the network may be a wired network or a wireless network.
[0036] The server 101 (also referred to as the data side) may be a server that provides various services, for example, a single server, a server cluster consisting of several servers, or a cloud computing service center, which is not limited in the present embodiment. The server 101 is provided with a first dimension table database, in which first dimension data is stored.
[0037] The business end 102 (also referred to as the business side) can be either hardware or software. When the business end 102 is hardware, it can be various electronic devices with a display screen that supports communication with the server end 101, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, etc.; when the business end 102 is software, it can be installed in the above electronic devices. The business end 102 can be implemented as multiple software or software modules, or as a single software or software module, and the embodiments of the present application do not limit this. A second dimension table database is provided in the business end 102, and the second dimension data is stored in the second dimension table database.
[0038] The first terminal 103 and the second terminal 104 can be hardware or software. Specifically, the first terminal 103 and the second terminal 104 can be vehicle terminals (the vehicle can be a smart vehicle such as a new energy vehicle with smart functions), or various electronic devices with display screens that support communication with the server 101. When the first terminal 103 and the second terminal 104 are software, they can be installed in the above electronic devices or vehicle terminals.
[0039] Figure 2 This is a schematic diagram of an automotive OTA upgrade strategy and task creation and delivery process provided by the embodiment of this application. Figure 2 The process of "upgrade strategy testing" is basically the same as the process of creating and issuing formal upgrade tasks. The difference is that the entire "upgrade strategy testing" is controlled by an automated test script.
[0040] The process of "upgrade strategy testing" is as follows:
[0041] In the first step, the automated test script automatically creates test tasks corresponding to the upgrade strategy and determines the test vehicles;
[0042] In the second step, the automated test script automatically generates corresponding test subtasks for each test vehicle based on the pre-configured test vehicle list, and sends each test subtask to the corresponding test vehicle;
[0043] In the third step, each test vehicle automatically completes the specified upgrade test according to the received test subtask and returns the test result.
[0044] In this example, after receiving the corresponding test subtask, the test vehicle executes the test subtask and continuously reports relevant information (data) of its execution stage to the cloud (such as server 101) during the entire test process. The reporting of these data is continuous, and the data reported at each time point can only represent its current status, so these data can be called streaming data.
[0045] After receiving the test results returned by each test vehicle, the cloud (server 101) will match the task information corresponding to the test subtask according to the task ID information carried by the test subtask corresponding to each test vehicle (if the test subtask data carries too much task information, it will cause a waste of data transmission resources). The task information here is usually unchanged or rarely changed, and can be called dimensional data.
[0046] It is understandable that in the formal upgrade task creation and delivery process, the upgrade results continuously reported by the target vehicle can be called streaming data. The upgrade results returned by the target vehicle are received by the cloud, and the task information corresponding to the upgrade subtask (these task information is usually unchanged or rarely changed) is matched according to the task ID information carried by the upgrade subtask corresponding to the target vehicle. This is called dimensional data.
[0047] The above streaming data and dimensional data can be associated by batch processing or streaming processing (such as Flink streaming processing). The process of associating streaming data and dimensional data by streaming processing is as follows: Figure 3 shown.
[0048] See also Figure 1~Figure 3 In the process of associating streaming data and dimensional data, it may be necessary to frequently read the dimensional data of the business end 102. In order to ensure the business functions and performance of the business end 102 and avoid affecting the business of the business end 102, the dimensional data of the business end 102 (i.e., the second dimensional data in the second dimensional database) can be synchronized to the first dimensional table database of the server end 101. Subsequently, when the server end 101 monitors and reads the streaming data to be associated sent by the first terminal 103, it can read the relevant first dimensional data from the first dimensional table database of the server end 101, thereby avoiding frequent reading of the dimensional data of the business end 102.
[0049] Since the first and second steps in the above-mentioned "upgrade strategy test" process are operated by automated test scripts, the creation and distribution of the test subtask (i.e., streaming data) and the generation and writing of the task data (i.e., dimensional data) are very close (usually only a few to tens of milliseconds apart). In addition, since it is necessary to first execute the data synchronization process between the second dimensional database of the business end 102 and the first dimensional table database of the server end 101, the following problem may occur: when the relevant information (streaming data) of the test subtask returned by the first terminal 103 (such as the test vehicle) reaches the server end 101, if the first dimensional table database of the server end 101 fails to complete the synchronization of the first dimensional data of the relevant information of the test subtask in time, then the test subtask and its corresponding task dimension information will fail to be associated and thus be empty. When counting data, these streaming data (such as OTA (Over the Air Technology, air download technology) upgrade subtasks) that fail to be associated with the task dimension information (dimensional data) will be missed, resulting in errors in the statistical results.
[0050] In order to solve the above technical problems, an embodiment of the present application proposes a delayed association method for streaming data and dimensional data, by creating a connection pool and a dimensional data cache object corresponding to a first dimensional table database on the server side 101, wherein the first dimensional data of the first dimensional table database is synchronized with the second dimensional data of the second dimensional table database on the business side 102; the server side 101 monitors and reads the current streaming data to be associated sent by the first terminal 103, and records the data reading timestamp and data type identifier of the current streaming data to be associated; if it is determined that the current streaming data to be associated is consistent with the dimensional data cache object and the first dimensional table database, If the association fails and the current streaming data to be associated meets the preset waiting and retry conditions, the retry waiting time after the current streaming data to be associated enters the waiting and retry mode is determined; if the retry waiting time reaches the current latest waiting time, the association operation between the current streaming data to be associated and the first dimension table database is re-executed; if the current streaming data to be associated is successfully associated with the first dimension table database, the first target dimension data associated with the aforementioned streaming data to be associated is added to the dimension data cache object, and the target streaming data is saved and output, and the target streaming data is composed of the current streaming data to be associated and its associated first target dimension data. The technical solution of the embodiment of the present application can be adapted to various data sources without making special modifications to the configuration of the data source, and without introducing additional message delay middleware, which can save additional middleware overhead, and can effectively solve the problem of errors in subsequent data statistical results caused by the failure to associate streaming data with dimension data.
[0051] It should be noted that the specific types, quantities and combinations of the server 101, the business end 102, the first terminal 103 and the second terminal 104 can be adjusted according to the actual needs of the application scenario, and the embodiments of the present application do not limit this.
[0052] Figure 4 It is a flowchart of a method for delayed association of streaming data and dimensional data provided in an embodiment of the present application. Figure 4 The delayed association method of streaming data and dimensional data can be obtained by Figure 1 The server 101 executes. Figure 4 As shown, the delayed association method of the streaming data and the dimensional data may specifically include the following steps:
[0053] Step S401: Create a connection pool and a dimension data cache object corresponding to a first dimension table database, and synchronize first dimension data of the first dimension table database with second dimension data of a second dimension table database at the business end.
[0054] The connection pool (which can be referred to as ConnectionPool) includes a connection object, which is the basic component of the connection pool and represents a connection to the first-dimensional table database (which can be referred to as Connection). The connection object contains connection parameters and status information, such as the first-dimensional table database URL (Universal Resource Locator), user name, password, connection status, etc. The connection object can be shared by multiple applications, thus avoiding the system overhead and performance problems caused by frequently creating and closing connection objects.
[0055] The configuration items of the dimensional data cache object (DimCache) mainly include the initial cache size, maximum cache size, cache validity period, etc.
[0056] Before executing the task of associating streaming data with dimensional data, the server 101 may first perform an initialization step. During the initialization process, a connection pool and a dimensional data cache object corresponding to the first dimensional table database are created.
[0057] It should be noted that the data synchronization task of the first dimension data of the first dimension table database and the second dimension data of the second dimension table database of the business end 102 performed by the server end 101 is independent of the association task between the current streaming data to be associated and the first dimension table data in the first dimension table database sent by the server end 101 to the first terminal 103, and they do not interfere with each other. In other words, the above-mentioned association task is triggered only when it is detected that the current streaming data to be associated arrives at the server end 101; and the execution of the above-mentioned data synchronization task is not affected by whether the above-mentioned association task is executed or not.
[0058] As a task that requires continuous data reading, data processing, and data output, the data synchronization task usually only needs to be started once in the entire task life cycle, and then it will remain active until the synchronization service is no longer needed. Therefore, some functions that only need to be executed once, and some content that only needs to be executed at initialization, can be placed in the open operator for execution when the data synchronization task is started.
[0059] Step S402: monitor and read the current streaming data to be associated sent by the first terminal, and record the data reading timestamp and data type identifier of the current streaming data to be associated.
[0060] The current streaming data to be associated may be information related to the task execution phase reported by the first terminal 103 to the server 101 at the current time point.
[0061] The data reading timestamp (which may be referred to as inputTimestamp) refers to the timestamp when the server 101 reads the current streaming data to be associated.
[0062] The data type identifier (which can be represented by isSnapshot) is used to indicate whether the current streaming data to be associated is snapshot data or non-snapshot data.
[0063] Streaming data is specific behavioral data. All subsequent data processing operations will be triggered by the arrival of streaming data at the server. Dimensional data is used to supplement some dimensions of streaming data. When no streaming data arrives, the server does not need to actively monitor changes in dimensional data.
[0064] After creating the connection pool and dimension data cache object corresponding to the first dimension table database, the server starts to monitor and read the data source of the streaming data (such as the data source of the first terminal 103).
[0065] You can use MySQL's cdc mode monitoring and / or Kafka data monitoring to monitor the data source of streaming data. Among them, MySQL CDC (Change Data Capture) is a technology that intercepts binlog in MySQL master-slave replication stream, thereby capturing the addition, deletion, and modification operations in the database in real time.
[0066] Step S403, if it is determined that the current streaming data to be associated fails to be associated with the dimension data cache object and the first dimension table database, and the current streaming data to be associated meets the preset waiting and retry conditions, then determine the retry waiting time after the current streaming data to be associated enters the waiting and retry mode.
[0067] Step S404: if the retry waiting time reaches the current latest waiting time, the association operation between the current streaming data to be associated and the first dimension table database is re-executed.
[0068] The retry waiting time is used to indicate how long the current streaming data to be associated needs to wait after entering the waiting retry mode before the association operation between the current streaming data to be associated and the first dimension table database can be re-executed.
[0069] The current latest waiting time can be a fixed waiting time or a dynamic waiting time.
[0070] Fixed waiting time means that the retry waiting time of each stream data to be associated that enters the waiting retry mode remains unchanged. For example, if the user specifies a fixed waiting time of 3 seconds (the minimum waiting time and the maximum waiting time are both 3 seconds, which can be expressed as (3s, 3s)), then the retry waiting time of the current stream data to be associated that enters the waiting retry mode is 3 seconds, and the retry waiting time of the next stream data to be associated after the current stream data to be associated is also 3 seconds, and the retry waiting time of the next stream data to be associated after the next stream data to be associated is also 3 seconds.
[0071] The dynamic waiting time means that the retry waiting time for each piece of streaming data to be associated that enters the waiting retry mode is not fixed, but can be dynamically adjusted according to the actual situation.
[0072] Step S405, if the current streaming data to be associated is successfully associated with the first dimension table database, the first target dimension data associated with the aforementioned streaming data to be associated is added to the dimension data cache object, and the target streaming data is saved and output. The target streaming data consists of the current streaming data to be associated and its associated first target dimension data.
[0073] The technical solution provided in the embodiment of the present application can adapt to various data sources without making special modifications to the configuration of the data source, and without introducing additional message delay middleware, thereby saving additional middleware overhead, and can effectively solve the problem of errors in subsequent data statistical results caused by the failure to associate streaming data with dimensional data.
[0074] In some embodiments, if the current streaming data to be associated fails to be associated with both the dimension data cache object and the first dimension table database, and the current streaming data to be associated meets a preset waiting retry condition, determining the retry waiting time after the current streaming data to be associated enters the waiting retry mode includes:
[0075] Determine whether the current streaming data to be associated is successfully associated with the dimension data cache object;
[0076] If it is determined that the current streaming data to be associated fails to be associated with the dimension data cache object, then it is determined whether the current streaming data to be associated is successfully associated with the first dimension table database;
[0077] If it is determined that the current streaming data to be associated fails to be associated with the first dimension table database, then determining whether the current streaming data to be associated meets a preset waiting retry condition;
[0078] If it is determined that the current streaming data to be associated meets the preset waiting and retry condition, then the retry waiting time after the current streaming data to be associated enters the waiting and retry mode is determined.
[0079] Specifically, first, it can be determined whether there is cached dimension data corresponding to the current streaming data to be associated in the dimension data cache object (DimCache) (usually the field content that must be associated with the current streaming data to be associated). If there is no cached dimension data corresponding to the current streaming data to be associated in the dimension data cache object, it can be determined that the current streaming data to be associated has failed to be associated with the dimension data cache object.
[0080] Then, it can be determined whether the first target dimension data corresponding to the current stream data to be associated exists in the first dimension table database. If the first dimension table database does not exist in the first target dimension data corresponding to the current stream data to be associated, it can be determined that the current stream data to be associated fails to be associated with the first dimension table database.
[0081] Next, it is determined whether the current streaming data to be associated meets the preset waiting and retrying conditions. The preset waiting and retrying conditions can adopt a user-defined rule engine mechanism, allowing users to define data processing rules according to their actual needs. The server can dynamically adjust the data processing and association process according to these data processing rules, thereby improving the flexibility of the system and user satisfaction.
[0082] The relevant configuration items of the data processing rules in the embodiments of the present application include but are not limited to: retry times threshold (max_retry_times), time difference threshold (max_time_diff), retry wait time interval range (minimum wait time min_wait_time, maximum wait time max_wait_time) and retry mode (including ordinary mode ordinary_mode and strict mode strict_mode), etc.
[0083] In some embodiments, determining whether the current streaming data to be associated meets a preset waiting retry condition includes:
[0084] Determine whether the current streaming data to be associated is snapshot data based on the data type identifier;
[0085] If the current streaming data to be associated is non-snapshot data, determining whether the number of waiting retries for the current streaming data to be associated to enter the waiting retry mode exceeds a preset number threshold;
[0086] If the number of waiting retries does not exceed the preset number threshold, the time difference between the data reading timestamp of the current streaming data to be associated and the current system timestamp is calculated;
[0087] Determine whether the time difference is less than or equal to a preset time difference threshold;
[0088] If the time difference is less than or equal to the preset time difference threshold, it is determined that the current streaming data to be associated meets the preset waiting retry condition.
[0089] Figure 5 This is a flow chart of judging whether the current streaming data to be associated meets the preset waiting and retrying condition provided by the embodiment of the present application. For ease of understanding, only part of the content related to this embodiment is shown in the figure.
[0090] See also Figure 5 Step S501: First, determine whether the current streaming data to be associated is snapshot data. Specifically, the data type identifier (isSnapshot) carried by the current streaming data to be associated can be extracted, and whether the current streaming data to be associated is snapshot data can be determined according to the data type identifier.
[0091] Step S502: If the current streaming data to be associated is non-snapshot data (isSnapshot=false), it is determined whether the number of waiting retries for the current streaming data to be associated to enter the waiting retry mode exceeds a preset number threshold. The preset number threshold may be a user-defined retry number threshold (max_retry_times), for example, the retry number threshold may be set to 3 times, 4 times, etc., and the embodiment of the present application does not impose specific restrictions on this.
[0092] Step S503: if the current streaming data to be associated is snapshot data (isSnapshot=true), then exit the loop. That is, the current streaming data to be associated does not meet the preset waiting and retrying condition, does not enter the waiting and retrying mode, and will be directly output.
[0093] Step S504: If the number of waiting retries of the current streaming data to be associated in the waiting retry mode does not exceed the preset number threshold (less than or equal to the preset number threshold), the time difference between the data reading timestamp (inputTimestamp) of the current streaming data to be associated and the current system timestamp is calculated. The current system timestamp can be directly obtained from the system time.
[0094] If the number of waiting retries for the current streaming data to be associated to enter the waiting retry mode exceeds the preset number threshold, jump to step S503.
[0095] Step S505, determine whether the time difference is less than or equal to a preset time difference threshold. The preset time difference threshold may be a user-defined time difference threshold (max_time_diff), for example, the time difference threshold may be set to 3 seconds, 5 seconds, etc., which is not specifically limited in the present embodiment.
[0096] Step S506: If the time difference is less than or equal to the preset time difference threshold, it is determined that the current streaming data to be associated meets the preset waiting retry condition, and the waiting retry mode is entered.
[0097] If the time difference is greater than the preset time difference threshold, jump to step S503.
[0098] The preset waiting and retry conditions include: ① The current streaming data to be associated is non-snapshot data; ② The number of waiting and retry attempts for the current streaming data to be associated to enter the waiting and retry mode does not exceed the preset number threshold; ③ The time difference between the data reading timestamp of the current streaming data to be associated and the current system timestamp is less than or equal to the preset time difference threshold. Only when conditions ①~③ are met, the current streaming data to be associated meets the preset waiting and retry conditions and enters the waiting and retry mode. If at least one of conditions ①~③ is not met, the current streaming data to be associated does not meet the preset waiting and retry conditions and will be directly output.
[0099] In some embodiments, if the retry waiting time reaches the current latest waiting time, before re-executing the association operation between the current streaming data to be associated and the first dimension table database, the method further includes:
[0100] Get the minimum preset waiting time and maximum preset waiting time for the current streaming data to enter the waiting retry mode;
[0101] Count the overall retry success rate of n historical streaming data to be associated; where the historical arrival time of the n historical streaming data to be associated arriving at the server is earlier than the current arrival time of the current streaming data to be associated arriving at the server, and n is an integer ≥ 0;
[0102] The current latest waiting time is determined based on the minimum preset waiting time, the maximum preset waiting time, and the overall retry success rate.
[0103] In some embodiments, determining the current latest waiting time according to the minimum preset waiting time, the maximum preset waiting time and the overall retry success rate includes:
[0104] Determine the weight coefficient corresponding to the overall retry success rate;
[0105] Get the last latest waiting time, and calculate the current waiting time based on the last latest waiting time, the overall retry success rate and the weight coefficient;
[0106] If the current required waiting time is greater than or equal to the minimum preset waiting time and less than or equal to the maximum preset waiting time, the current required waiting time is determined as the current latest waiting time;
[0107] If the current waiting time is less than the minimum preset waiting time, or greater than the maximum preset waiting time, the last latest waiting time is determined as the current latest waiting time.
[0108] The minimum preset waiting time can be the lowest waiting time in the retry waiting time interval range set by the user. min_wait_time The maximum preset waiting time can be the highest waiting time in the retry waiting time interval range set by the user ( max_wait_time ).
[0109] Figure 6 This is a schematic diagram of a process for determining the current latest waiting time provided by an embodiment of the present application. For ease of understanding, only some contents related to the embodiment of the present application are shown in the figure.
[0110] See also Figure 6 , step S601, initialization waiting time ( wait_time ),in, .
[0111] Step S602: If there is new streaming data to be associated that meets the preset waiting retry condition, then go to step S603.
[0112] Step S603, counting the overall retry success rate of n historical streaming data to be associated ( success_rate ).in, ; m represents the number of successfully retried items among the n historical streaming data to be associated. The n historical streaming data to be associated are the streaming data that arrived at the server before the current streaming data to be associated arrived at the server and completed the waiting for retry process. For example, the current streaming data to be associated is the 100th streaming data that arrived at the server and entered the waiting for retry process, where the n historical streaming data to be associated can be any n of the 1st to 99th streaming data that arrived at the server before the current streaming data to be associated arrived at the server and completed the waiting for retry process. In this example, the value range of n is 0 to 99.
[0113] Step S604, determine the weight coefficient corresponding to the overall retry success rate of the n historical streaming data to be associated. The weight number can be customized by the user according to the actual situation, or determined by the system according to a preset "retry success rate-weight coefficient" correspondence table.
[0114] Exemplarily, a preset “retry success rate-weight coefficient” correspondence table is shown in Table 1.
[0115] Table 1 Correspondence table of "retry success rate-weight coefficient"
[0116]
[0117] Among them, the value range of W1~W4 can be 0 (excluding 0)~1 (including 1).
[0118] Get the last latest waiting time, and calculate the current waiting time based on the last latest waiting time, the overall retry success rate and the weight coefficient ( wait_time_new ).
[0119] Specifically, the current waiting time can be calculated according to the following formula (1): wait_time_new ).
[0120] (1);
[0121] In formula (1), wait_time_new Indicates the current waiting time; wait_time_old Indicates the last latest waiting time; success_rate Indicates the overall retry success rate of n historical streaming data to be associated; w Represents the weight coefficient. success_rate After that, you can query the above table 1 to determine the Corresponding w .
[0122] Step S605, determine the current waiting time ( wait_time_new ) is within the retry waiting time interval, that is, to determine the current waiting time ( wait_time_new ) is greater than or equal to the minimum preset waiting time and less than or equal to the maximum preset waiting time.
[0123] Step S606: If the current waiting time is ( wait_time_new ) is not within the retry waiting time interval, the last latest waiting time is determined as the current latest waiting time. wait_time=wait_time_old (constant).
[0124] Step S607, if the current waiting time is required ( wait_time_new) is within the retry waiting time interval, the current waiting time is determined as the current latest waiting time. wait_time=wait_time_new .
[0125] Step S608, continue to wait for the arrival of other streaming data to be associated and process processing.
[0126] It can be understood that after the current streaming data to be associated is processed and the retry result is returned to the server, the server recalculates the current latest waiting time according to the above steps S601 to S608, and the current latest waiting time is used as the retry waiting time for the next streaming data to be associated.
[0127] Through the above method, the retry waiting time of the next streaming data to be associated that enters the waiting retry mode can be dynamically adjusted according to the retry success rate and weight coefficient of the historical streaming data to be associated that enters the waiting retry mode (including the current streaming data to be associated). This can not only improve the efficiency of data processing, but also help to improve the success rate of associating streaming data with dimensional data.
[0128] If the retry waiting time of the current streaming data to be associated reaches the current latest waiting time, the retry process is entered, and the server 101 re-executes the association operation between the current streaming data to be associated and the first dimension table database.
[0129] In some embodiments, determining that the current streaming data to be associated fails to be associated with the first dimension table database includes:
[0130] Check whether the connection between the first dimension table database and the connection pool is in a maintained connection state;
[0131] If the connection between the first dimension table database and the connection pool is in a disconnected state, reestablish the connection between the first dimension table database and the connection pool;
[0132] When the connection between the first dimension table database and the connection pool is reestablished, the association field between the current streaming data to be associated and the first dimension table database is obtained;
[0133] Querying whether there is first target dimension table data corresponding to the associated field in the first dimension table database;
[0134] If the first target dimension table data corresponding to the associated field does not exist in the first dimension table database, it is determined that the current streaming data to be associated fails to be associated with the first dimension table database.
[0135] Specifically, the server 101 first checks whether the connection (Connection) between the first dimension table database and the connection pool (ConnectionPool) created during initialization is in a maintained connection state. If the connection between the first dimension table database and the connection pool is in a disconnected state, then re-acquire the connection from the connection pool ConnectionPool, that is, re-establish the connection between the first dimension table database and the connection pool. When the connection between the first dimension table database and the connection pool is re-established, obtain the associated field that the current streaming data to be associated needs to be associated with the first dimension table database (for example, the dimension table in the first dimension table database is a task table, and the condition for associating the streaming data with the dimension data may be: streamData.taskUuid = dimData.uuid); then, query whether there is the first target dimension table data corresponding to the associated field in the first dimension table database. If the first target dimension table data corresponding to the associated field is queried in the first dimension table database, it can be determined that the current streaming data to be associated is successfully associated with the first dimension table database. At this time, the first target dimension table data can be filled into the data stream of the current streaming data to be associated; at the same time, the result of the successful match (association) is placed in the dimension data cache object (DimCache) so that the dimension information can be directly read from the cache within a period of time.
[0136] If the first target dimension table data corresponding to the associated field is not found in the first dimension table database, it can be determined that the current streaming data to be associated has temporarily failed to be associated with the first dimension table database, and it is necessary to further determine whether the current streaming data to be associated meets the preset waiting retry condition.
[0137] In some embodiments, determining the retry waiting time after the current streaming data to be associated enters the waiting retry mode includes:
[0138] If it is determined that the waiting retry mode is the normal mode, the current streaming data to be associated is pushed from the original stream to the side output stream;
[0139] Calculates the retry waiting time of the current streaming data to be associated in the side output stream waiting for retry.
[0140] In streaming data processing, an important concern is timing. In some cases, the order of data output must be the same as the order of input, otherwise some result data exceptions will be caused. In this case, it is necessary to strictly ensure that the output order of data is consistent with the input order. At this time, the strict mode can be used to perform associated waits and retries to ensure that the order of data is not affected.
[0141] Strict mode and non-strict mode (non-strict mode is also called normal mode) can be specified by the user through self-configuration according to the user's actual usage scenarios and needs.
[0142] Figure 7 It is a comparative schematic diagram of the data processing flow of a strict mode and a normal mode provided in an embodiment of the present application.
[0143] See also Figure 7 In strict mode, each record in the data stream (for example, data a, b, c) performs associated operations in sequence. When a record enters the waiting state, all subsequent data enter the waiting state at the same time. After the previous record is processed, the next record is processed.
[0144] In the embodiment of the present application, three waiting retry determination conditions are introduced (such as Figure 5 The waiting and retry mode will not be entered until the associated streaming data meets the three waiting and retry judgment conditions at the same time, which can ensure that the overall waiting time of the entire data stream will not exceed "retry interval * number of retries".
[0145] As an example, suppose there are 10 stream data D1~D10 to be associated, and the reading time is 10:00:01~10:00:10 respectively. Assume that the stream data D2 to be associated fails to be associated with both the dimension data cache object and the first dimension table database, then it will enter the waiting and retry process. Assume that after the stream data D2 to be associated waits for 3 seconds twice, the time difference between the data receiving time (data reading timestamp) and the current time (current system timestamp) is greater than the time difference threshold (such as 5 seconds), then D2 will be directly output because it no longer meets the three waiting and retry judgment conditions at the same time. At this time, if the subsequent D3 fails to be associated with both the dimension data cache object and the first dimension table database, then the time difference between its data receiving time and the current time will be significantly greater than the time difference threshold (such as 5 seconds) due to the 6 seconds of waiting of D2. At this time, although D3 fails to match, it still cannot meet the above three waiting and retry judgment conditions at the same time, and will be directly output without adding the waiting time.
[0146] Therefore, if continuous association failures occur, usually only the previous streaming data to be associated that simultaneously meets the above three waiting-for-retry judgment conditions will enter the waiting-for-retry mode, while the streaming data to be associated after this streaming data to be associated will not enter the waiting-for-retry mode because it cannot simultaneously meet the above three waiting-for-retry judgment conditions.
[0147] In some cases, the order of data is not so strict, that is, the target data has no requirements on the order, or there is a function of order correction in the subsequent steps. The waiting retry mode can adopt the normal (non-strict) mode, which can make the delay of data waiting have less impact on normal data and improve data processing efficiency.
[0148] In this case, some diversion methods are used to divert the stream data records that need to be waited for to be processed into separate processing steps. For example, Flink's side output stream. At this time, the streaming data to be associated that needs to wait for retry will enter the side output stream from the original stream in sequence, and wait for retry and other operations will be performed in the logic of the side output stream. Normal streaming data that does not need to wait for retry can perform subsequent operations normally in the original stream. The waiting of the streaming data to be associated in the side output stream will not affect the normal streaming data. The two do not interfere with each other, which can make data processing more efficient.
[0149] If the current streaming data to be associated meets the preset waiting retry condition, the waiting retry mode is entered, the current streaming data to be associated is pushed from the original stream to the side output stream, and the retry waiting time is calculated.
[0150] In some embodiments, monitoring and reading the current to-be-associated streaming data sent by the first terminal includes:
[0151] Record the log location information at the current time point, and set a read lock for the streaming data table corresponding to the current streaming data to be associated;
[0152] Read the existing streaming data in the streaming data table in the form of a snapshot, and record the data reading timestamp and data type identifier of the existing streaming data;
[0153] After reading is completed, release the read lock of the streaming data table;
[0154] Starting from the log position information, the changed streaming data of the streaming data table is read in sequence, and the data reading timestamp and data type identifier of the changed streaming data are recorded; wherein the current streaming data to be associated is the existing streaming data or the changed streaming data.
[0155] If the data source of the current streaming data to be associated is stored in a MySQL database, the server 101 can use a snapshot + incremental change record method to monitor and read the streaming data sent by the first terminal 103. If only the incremental change record (binlog in MySQL) is read, historical data in the streaming data table will be missed, because the binlog stores the change records, and the data before the existing binlog cannot be read through the binlog.
[0156] Full synchronization usually takes the form of snapshots, which are complete data backups at a certain point in time.
[0157] First, the log position information (i.e., binlog position information) at the current time point is recorded, and then a read lock is applied to the currently monitored streaming data table (i.e., the streaming data table corresponding to the current streaming data to be associated). The lock exists to ensure that the data will not be updated during the snapshot creation. If the requirement for "exactly once" is not strict and "at least once" is acceptable, no read lock can be applied to the streaming data table.
[0158] Binlog is a binary file that records SQL statements used by users to update the database. For example, SQL statements that change database tables and content will be recorded in binlog, but queries on database tables and other content will not be recorded.
[0159] Then, create a snapshot, read all existing streaming data in the streaming data table, and record the data reading timestamp and data type identifier of each existing streaming data. For example, in Flink's mysql-cdc connector, sourceRecord.sourceOffset.isSnapshot can be used to mark whether this streaming data is snapshot data. If you use other tools or implement data stream reading yourself, you can record whether the data is a snapshot by consulting the tool documentation or implementing related functions yourself.
[0160] After the reading is completed, the read lock of the streaming data table is released, allowing the streaming data table to be updated.
[0161] Binlog is one of the core features of MySQL. It is used to record all modification operations performed on the database (excluding operations such as SELECT and SHOW), including data insertion, update, and deletion operations.
[0162] Starting from the binlog location information recorded in the above steps, read the binlog logs in sequence. Each line of the log corresponds to a change in the data, so each line of data is read in sequence as a line of streaming data. In this way, the changed streaming data of the streaming data table can be read in sequence. At the same time, the data reading timestamp and data type identifier of the read changed streaming data are recorded.
[0163] If the currently monitored streaming data table is a data queue such as Kafka, its data does not need to be snapshotted and can be read sequentially. In this case, the data type of each piece of data is incremental data rather than snapshot data. At the same time, the data reading timestamp and data type identifier corresponding to each piece of streaming data read are recorded.
[0164] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present application, which will not be described one by one here.
[0165] Figure 8 It is a flowchart of another delayed association method of streaming data and dimensional data provided by an embodiment of the present application. The flowchart does not include task initialization and pre-task exit operations.
[0166] See also Figure 8 , step S801, listen to and read the current streaming data to be associated sent by the first terminal, and record the data reading timestamp (inputTimestamp) and data type identifier (isSnapshot) of the current streaming data to be associated. This step includes verifying the data receiving time (i.e., data reading timestamp) of the current streaming data to be associated and determining whether the current streaming data to be associated is snapshot data.
[0167] Step S802, determining whether the current streaming data to be associated is successfully associated with the dimension data cache object.
[0168] Step S803: if the current streaming data to be associated fails to be associated with the dimension data cache object, the first dimension table database is queried to perform an association operation between the current streaming data to be associated and the first dimension table database.
[0169] If the current streaming data to be associated is successfully associated with the dimension data cache object, jump to step S808.
[0170] Step S804, determining whether the current stream data to be associated is successfully associated with the first dimension table database.
[0171] If the current streaming data to be associated fails to be associated with the first dimension table database, the process jumps to step S805.
[0172] If the current streaming data to be associated is successfully associated with the first dimension table database, jump to step S807.
[0173] Step S805 , determining whether the current streaming data to be associated meets a preset waiting retry condition (data rule processing engine retry condition).
[0174] Step S806: If the current streaming data to be associated meets the preset waiting and retrying condition, the waiting and retrying mode is entered and the process returns to step S803.
[0175] If the current streaming data to be associated does not meet the preset waiting and retrying condition, jump to step S808.
[0176] Step S807: adding the first target dimension data associated with the current streaming data to be associated to the dimension data cache object.
[0177] Step S808, data outflow, that is, saving and outputting the target streaming data, the target streaming data consists of the current streaming data to be associated and its associated first target dimension data.
[0178] The target streaming data finally generated in the above steps can be collected by the data output operator, packaged into the format required by the target database, and written to the target database in the form of stream or batch.
[0179] The target streaming data may be output from the server 101 to the second terminal 104 .
[0180] In order to further illustrate the technical solution of the embodiment of the present application, a specific implementation method is listed below for detailed description.
[0181] Taking MySQL-cdc as an example, assume that the streaming data comes from the MySQL subtask table subtaskTable. The dimension data that the subtask needs to be associated with comes from the MySQL task table taskTable. Assume that the user specifies the following configuration items in the configuration: the waiting interval is 3s (that is, configure max_wait_time = min_wait_time = 3s), the time difference threshold is 5s, the retry threshold is 3 times, and the retry mode is strict mode.
[0182] Assume that before the data synchronization service is started, the table structure and existing data of the subtaskTable table are as shown in Table 2 below, and the table structure and existing data of the taskTable table are as shown in Table 3 below.
[0183] Table 2
[0184]
[0185] Table 3
[0186]
[0187] Subsequent streaming data comes from changes in the subtaskTable table.
[0188] The target data is all data changes in the subtaskTable table, which is associated with the taskTable table. The condition for association is: subtaskTable.taskUuid = taskTable.uuid.
[0189] And add the associated taskTable.taskName to the streaming data, and finally write it to the target database.
[0190] (1) Task start and initialization:
[0191] When the associated task is started, the server creates the following content:
[0192] 1) Create a connection pool (ConnectionPool) for the first dimension table database. Get a new connection object (Connection) from the connection pool.
[0193] 2) Create and configure the dimension data cache object (DimCache).
[0194] At this point, DimCache is empty.
[0195] (2) Processing of snapshot data:
[0196] The server reads three existing data in the subtaskTable table in a snapshot streaming manner and adds two attributes, isSnapshot and inputTimestamp, to these three data. The specific values are shown in a table as shown in Table 4 below.
[0197] Table 4
[0198]
[0199] The following describes the processing process and results of these three data respectively in the form of streaming data processing one by one.
[0200] ① The processing process of streaming data subtask1 is as follows:
[0201] The value of the field used to associate this data (streaming data subtask1) with the dimensional data is taskUuid =taskA.
[0202] The processing time of the current data is 10:10:01.
[0203] The value taskA is searched in DimCache. Because the synchronization service has just started, DimCache is empty at this time, so this step of search fails, that is, the streaming data subtask1 fails to be associated with DimCache.
[0204] In the dimension table taskTable of the first dimension table database, search for the first dimension data with Uuid = taskA. As we know from the previous text, there is only one data in taskTable at this time, which is the data with Uuid = taskA. At this time, take out the required taskName value, that is, taskName = taskA.
[0205] On the one hand, taskName is added to the streaming data subtask1 for data output. At this time, the output data structure and content are shown in Table 5 below.
[0206] Table 5
[0207]
[0208] On the other hand, the first dimension data is written into the cache DimCache, and the content of DimCache becomes as shown in Table 6 below.
[0209] Table 6
[0210]
[0211] ② The processing of streaming data subtask2 is as follows:
[0212] The value of the field used to associate this data (streaming data subtask2) with the dimensional data is taskUuid =taskB.
[0213] The processing time of the current data is 10:10:01.
[0214] Failed to find the value of taskUuid = taskB in DimCache, go to the next step.
[0215] In the dimension table taskTable of the first dimension table database, the value of uuid = taskB cannot be found, so proceed to the next step.
[0216] At this time, since the query of cache and dimension database fails, it is judged whether the streaming data subtask2 meets the preset waiting and retry conditions (i.e., enters the Figure 5 Since this data is snapshot data and does not meet the preset waiting and retrying conditions, it directly jumps to the data output step. At this time, the output data structure and content are shown in Table 7 below.
[0217] Table 7
[0218]
[0219] ③ The processing of streaming data subtask3 is as follows:
[0220] The value of the field used to associate this data (streaming data subtask3) with the dimensional data is taskUuid =taskA.
[0221] The processing time of the current data is 10:10:01.
[0222] The corresponding cache dimension data is searched in DimCache, and the cache dimension data is added to the streaming data subtask3 and output. The output data structure and content are shown in Table 8 below.
[0223] Table 8
[0224]
[0225] (3) Processing of incremental data:
[0226] The changes in the subtaskTable table are recorded in the MySQL binlog file and read by Flink streaming. When reading these incremental data, two attributes are added to these incremental data: isSnapshot and inputTimestamp. The specific values are shown in table form as shown in Table 9 below.
[0227] Table 9
[0228]
[0229] At the same time, when the data in the subtaskTable table is changed, the dimension table taskTable table also undergoes some data changes. The specific change time and value are shown in the following Table 10. Among them, the dataTime field indicates the time when the row is written into the table, that is, before this time point, this data cannot be queried in the dimension table.
[0230] Table 10
[0231]
[0232] The following describes the processing process and results of these data in the form of processing streaming data one by one.
[0233] ④ The processing of streaming data subtask4 is as follows:
[0234] The value of the field used to associate this data (streaming data subtask4) with the dimensional data is taskUuid =taskC.
[0235] The processing time of the current data is 10:10:02.
[0236] Failed to find the value of taskUuid = taskC in DimCache, go to the next step.
[0237] In the dimension table taskTable of the first dimension table database, the value of uuid = taskC is found, added to the streaming data subtask4, and output. At this time, the output data structure and content are shown in Table 11 below.
[0238] Table 11
[0239]
[0240] At the same time, this dimension data is written into the cache DimCache, and the content of DimCache becomes as shown in Table 12 below.
[0241] Table 12
[0242]
[0243] ⑤ The processing of streaming data subtask5 is as follows:
[0244] The value of the field used to associate this data with the dimension data is taskUuid = taskC.
[0245] The processing time of the current data is 10:10:03.
[0246] The corresponding cache dimension data is searched in DimCache, added to the streaming data subtask5, and output. The output data structure and content are shown in Table 13 below.
[0247] Table 13
[0248]
[0249] ⑥ The processing of streaming data subtask6 is as follows:
[0250] The value of the field used to associate this data with the dimension data is taskUuid = taskD.
[0251] The processing time of the current data is 10:10:04.
[0252] At this time, the dimension table data is shown in Table 14 below.
[0253] Table 14
[0254]
[0255] Similar to the previous example, when streaming data subtask6 arrives at the server (10:10:04), the server queries DimCache and the first dimension table database, but fails to find the dimension data associated with streaming data subtask6. That is, streaming data subtask6 fails to associate with DimCache and the first dimension table database. Further, it is determined whether streaming data subtask6 meets the preset waiting and retrying conditions (for the specific process, please refer to Figure 5 ).
[0256] After judgment, it is known that streaming data subtask6 meets the following three conditions at the same time: 1) it is non-snapshot data; 2) the number of retries is less than or equal to 3 times; 3) the time difference between the current system timestamp (currentTimestamp) and the data reading timestamp (inputTimestamp) is less than or equal to 5 seconds. Therefore, it enters a 3s wait and retries after 3s (10:10:07).
[0257] At this time, the dimension table data is shown in Table 15 below.
[0258] Table 15
[0259]
[0260] At this time, the data of uuid = taskD in the dimension table has been updated (its update time is 10:10:05). Therefore, in this retry, the dimension data is still not matched in DimCache, but the value of uuid = taskD is successfully found in the dimension table taskTable, added to the streaming data, and output. The output data structure and content are shown in Table 16 below.
[0261] Table 16
[0262]
[0263] At the same time, this dimension data is written into the cache DimCache, and the content of DimCache becomes as shown in Table 17 below.
[0264] Table 17
[0265]
[0266] ⑦ The processing of streaming data subtask7 is as follows:
[0267] The value of the field used to associate this data with the dimension data is taskUuid = taskE.
[0268] The time when streaming data subtask7 is received by the server is 10:10:05, but due to the waiting of streaming data subtask6, the current data processing time is 10:10:07 (that is, the time when streaming data subtask6 is processed). At this time, the data with uuid = taskE in the dimension table has been updated as shown in Table 18 below.
[0269] Table 18
[0270]
[0271] Failed to find the value of taskUuid = taskE in DimCache, go to the next step.
[0272] In the dimension table taskTable of the first dimension table database, the value of uuid = taskE is found and added to the streaming data for output. At this time, the output data structure and content are shown in Table 19 below.
[0273] Table 19
[0274]
[0275] ⑧The processing of streaming data subtask8 is as follows:
[0276] The value of the field used to associate this data with the dimension data is taskUuid = taskF.
[0277] The current data arrival time is 10:10:06, and the processing time is 10:10:07 (that is, the time when the streaming data subtask7 is processed). The dimension table data at this time is shown in Table 20 below.
[0278] Table 20
[0279]
[0280] The first query (10:10:07) of DimCache and the first dimension table database failed. We further determined whether the streaming data subtask8 met the preset waiting and retry conditions (i.e., entered the waiting and retrying state). Figure 5 rule determination steps shown).
[0281] After judgment, it is known that streaming data subtask8 meets the following three conditions at the same time: 1) it is non-snapshot data; 2) the number of retries is less than or equal to 3 times; 3) the time difference between the current system timestamp (currentTimestamp) and the data reading timestamp (inputTimestamp) is less than or equal to 5 seconds. Therefore, it enters a 3s wait and retries after 3s (10:10:10).
[0282] At this time, the dimension table data is shown in Table 21 below.
[0283] Table 21
[0284]
[0285] Retry: Query taskUuid = taskF in DimCache and the first dimension table database, both failed. Determine again whether the streaming data subtask8 meets the preset waiting retry conditions.
[0286] After judgment, it is known that streaming data subtask8 meets the following three conditions at the same time: 1) it is non-snapshot data; 2) the number of retries is less than or equal to 3 times; 3) the time difference between the current system timestamp (currentTimestamp) and the data reading timestamp (inputTimestamp) is less than or equal to 5 seconds. Therefore, it enters a 3s wait and retries after 3s (10:10:13).
[0287] At this time, the dimension table data is shown in Table 22 below.
[0288] Table 22
[0289]
[0290] Retry: Query taskUuid = taskF in DimCache and the first dimension table database, both failed. Determine again whether the streaming data subtask8 meets the preset waiting retry conditions.
[0291] After judgment, it can be seen that the streaming data subtask8 meets the following conditions 1) and 2) at the same time but does not meet condition 3): 1) it is non-snapshot data; 2) the number of retries is less than or equal to 3 times; 3) the time difference between the current system timestamp (currentTimestamp) and the data reading timestamp (inputTimestamp) is 7 seconds (exceeding the time difference threshold of 5 seconds). Therefore, this data jumps to the data output step, and the output data structure and content are shown in Table 23 below.
[0292] Table 23
[0293]
[0294] ⑨ The processing of streaming data subtask9 is as follows:
[0295] The value of the field used to associate this data with the dimension data is taskUuid = taskG.
[0296] The current data arrival time is 10:10:07, and the processing time is 10:10:13 (that is, the time when the streaming data subtask8 is processed). The dimension table data at this time is shown in Table 24 below.
[0297] Table 24
[0298]
[0299] Failed to find the value of taskUuid = taskG in DimCache, go to the next step.
[0300] In the dimension table taskTable of the first dimension table database, the value of uuid = taskG is found and added to the streaming data for output. At this time, the output data structure and content are shown in Table 25 below.
[0301] Table 25
[0302]
[0303] ⑩ The processing of streaming data subtask10 is as follows:
[0304] The value of the field used in this data to associate with the dimension data is taskUuid = taskH.
[0305] The current data arrival time is 10:10:07, and the processing time is 10:10:13 (that is, the time when the streaming data subtask9 is processed). The dimension table data at this time is shown in Table 26 below.
[0306] Table 26
[0307]
[0308] According to taskUuid = taskH, queries are made in DimCache and the first dimension table database, but both fail. Further judgment is made as to whether the streaming data subtask8 meets the preset waiting and retrying conditions (i.e., entering the Figure 5 rule determination steps shown).
[0309] After judgment, it can be seen that the streaming data subtask10 meets the following conditions 1) and 2) at the same time but does not meet condition 3): 1) it is non-snapshot data; 2) the number of retries is less than or equal to 3 times; 3) the time difference between the current system timestamp (currentTimestamp) and the data reading timestamp (inputTimestamp) is 6 seconds (exceeding the time difference threshold of 5 seconds). Therefore, this data jumps to the data output step, and the output data structure and content are shown in Table 27 below.
[0310] Table 27
[0311]
[0312] It can be seen from the above content that the maximum waiting time for all data will not exceed the result of the waiting time interval * the retry number threshold, which is 3s*3 times=9s in this example.
[0313] Compared with the traditional solution of using the built-in delay queue function of the middleware (only applicable to data sources that support delay queues), or the solution of using Flink's time window function (there is a large data storage overhead before the window is closed, and if a sliding window is used, it may also cause the problem of repeated data output), or the solution of adjusting data generation and consumption (which may involve changing the data source, increasing data buffering or adjusting the frequency of data release, etc., and in many cases involves coordinating with other systems to modify the implementation logic, which is difficult to implement), the technical solution provided by the present embodiment has at least the following advantages:
[0314] 1) It can be adapted to various data sources without making special modifications to the data source configuration or introducing additional message delay middleware.
[0315] 2) It will not cause duplicate data processing.
[0316] 3) No additional memory overhead will be incurred on data processing components.
[0317] 4) Only the streaming data that failed to be associated is processed additionally, normal data is not affected, and the delay time will not be accumulated.
[0318] 5) Specific delay conditions and processing solutions can be configured according to the actual needs of users, and are widely applicable.
[0319] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.
[0320] Fig. 9 Schematic diagram of a delayed association device for streaming data and dimensional data provided by an embodiment of the present application. Fig. 9 As shown, the delayed association device of the streaming data and the dimensional data may specifically include:
[0321] The creation module 901 is configured to create a connection pool and a dimension data cache object corresponding to the first dimension table database, and synchronize the first dimension data of the first dimension table database with the second dimension data of the second dimension table database of the business end;
[0322] The reading module 902 is configured to monitor and read the current streaming data to be associated sent by the first terminal, and record the data reading timestamp and data type identifier of the current streaming data to be associated;
[0323] The waiting module 903 is configured to determine the retry waiting time after the current streaming data to be associated enters the waiting retry mode if it is determined that the current streaming data to be associated fails to be associated with the dimension data cache object and the first dimension table database, and the current streaming data to be associated meets the preset waiting retry condition;
[0324] The retry module 904 is configured to re-execute the association operation between the current streaming data to be associated and the first dimension table database if the retry waiting time reaches the current latest waiting time;
[0325] The output module 905 is configured to add the first target dimension data associated with the current streaming data to be associated to the dimension data cache object if the current streaming data to be associated is successfully associated with the first dimension table database, save and output the target streaming data, and the target streaming data consists of the current streaming data to be associated and its associated first target dimension data.
[0326] In some embodiments, the waiting module 903 may include:
[0327] A first judgment unit is configured to judge whether the current streaming data to be associated is successfully associated with the dimension data cache object;
[0328] The second judgment unit is configured to judge whether the current streaming data to be associated is successfully associated with the first dimension table database if it is determined that the current streaming data to be associated fails to be associated with the dimension data cache object;
[0329] A third judgment unit is configured to judge whether the current streaming data to be associated meets a preset waiting retry condition if it is determined that the current streaming data to be associated fails to be associated with the first dimension table database;
[0330] The determining unit is configured to determine the retry waiting time after the current streaming data to be associated enters the waiting retry mode if it is determined that the current streaming data to be associated meets the preset waiting retry condition.
[0331] In some embodiments, the above-mentioned determination unit may include:
[0332] A first judgment component is configured to judge whether the current streaming data to be associated is snapshot data based on the data type identifier;
[0333] The second judgment component is configured to judge whether the number of waiting retries for the current streaming data to be associated to enter the waiting retry mode exceeds a preset number threshold if the current streaming data to be associated is non-snapshot data;
[0334] A calculation component is configured to calculate a time difference between a data reading timestamp of the current streaming data to be associated and a current system timestamp if the number of waiting retries does not exceed a preset number threshold;
[0335] A third judgment component is configured to judge whether the time difference is less than or equal to a preset time difference threshold;
[0336] The determination component is configured to determine that the current streaming data to be associated meets a preset waiting retry condition if the time difference value is less than or equal to a preset time difference threshold.
[0337] In some embodiments, the above device may further include:
[0338] An acquisition module is configured to acquire a minimum preset waiting time and a maximum preset waiting time for the current streaming data to be associated to enter a waiting retry mode;
[0339] A statistics module is configured to count the overall retry success rate of n historical streaming data to be associated; wherein the historical arrival time of the n historical streaming data to be associated arriving at the server is earlier than the current arrival time of the current streaming data to be associated arriving at the server, and n is an integer ≥ 0;
[0340] The determination module is configured to determine the current latest waiting time according to the minimum preset waiting time, the maximum preset waiting time and the overall retry success rate.
[0341] In some embodiments, the above-mentioned determination module may include:
[0342] A weight determination unit, configured to determine a weight coefficient corresponding to the overall retry success rate;
[0343] The calculation unit is configured to obtain the last latest waiting time, and calculate the current required waiting time according to the last latest waiting time, the overall retry success rate and the weight coefficient;
[0344] A first determining unit is configured to determine the current required waiting time as the current latest waiting time if the current required waiting time is greater than or equal to the minimum preset waiting time and less than or equal to the maximum preset waiting time;
[0345] The second determining unit is configured to determine the last latest waiting time as the current latest waiting time if the current required waiting time is less than the minimum preset waiting time, or greater than the maximum preset waiting time.
[0346] In some embodiments, the second determination unit may include:
[0347] A checking component is configured to check whether the connection between the first dimension table database and the connection pool is in a maintained connection state;
[0348] Establishing a component configured to reestablish the connection between the first dimension table database and the connection pool if the connection between the first dimension table database and the connection pool is in a disconnected state;
[0349] The field acquisition component is configured to acquire the associated field between the current streaming data to be associated and the first dimension table database when the connection between the first dimension table database and the connection pool is reestablished;
[0350] A query component is configured to query whether there is first target dimension table data corresponding to the associated field in the first dimension table database;
[0351] The association determination component is configured to determine that the current streaming data to be associated fails to be associated with the first dimension table database if the first target dimension table data corresponding to the associated field does not exist in the first dimension table database.
[0352] In some embodiments, the waiting module 903 may further include:
[0353] The push unit is configured to push the current streaming data to be associated from the original stream into the side output stream if it is determined that the waiting retry mode is the normal mode;
[0354] The duration calculation unit is configured to calculate the retry waiting duration of the current streaming data to be associated waiting for retry in the side output stream.
[0355] In some embodiments, the reading module 902 may include:
[0356] A recording unit is configured to record the log position information at the current time point and set a read lock for the streaming data table corresponding to the current streaming data to be associated;
[0357] A first reading unit is configured to read the existing streaming data in the streaming data table in a snapshot form, and record a data reading timestamp and a data type identifier of the existing streaming data;
[0358] A release unit is configured to release the read lock of the streaming data table after the reading is completed;
[0359] The second reading unit is configured to read the changed streaming data of the streaming data table in sequence starting from the log position information, and record the data reading timestamp and data type identifier of the changed streaming data; wherein the current streaming data to be associated is existing streaming data or changed streaming data.
[0360] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.
[0361] Fig.10 Schematic diagram of an electronic device 10 provided in an embodiment of the present application. Fig.10 As shown, the electronic device 10 of this embodiment includes: a processor 1001, a memory 1002, and a computer program 1003 stored in the memory 1002 and executable on the processor 1001. When the processor 1001 executes the computer program 1003, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor 1001 executes the computer program 1003, the functions of the modules / units in the above-mentioned device embodiments are implemented.
[0362] The electronic device 10 may be a desktop computer, a notebook, a PDA, a cloud server, or other electronic device. The electronic device 10 may include, but is not limited to, a processor 1001 and a memory 1002. Those skilled in the art will appreciate that Fig.10 The electronic device 10 is merely an example and does not limit the electronic device 10 . The electronic device 10 may include more or fewer components than shown in the figure, or different components.
[0363] The processor 1001 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.
[0364] The memory 1002 may be an internal storage unit of the electronic device 10, for example, a hard disk or memory of the electronic device 10. The memory 1002 may also be an external storage device of the electronic device 10, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card (FlashCard), etc. equipped on the electronic device 10. The memory 1002 may also include both an internal storage unit of the electronic device 10 and an external storage device. The memory 1002 is used to store computer programs and other programs and data required by the electronic device.
[0365] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units.
[0366] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium (such as a computer-readable storage medium). Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. The computer program may include computer program code, which may be in source code form, object code form, executable file or some intermediate form. The computer-readable storage medium may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.
[0367] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.
Claims
1. A delayed association method for streaming data and dimensional data, It is characterized in that include: Creating a connection pool and a dimension data cache object corresponding to a first dimension table database, wherein first dimension data of the first dimension table database is synchronized with second dimension data of a second dimension table database at the business end; Listening to and reading the current streaming data to be associated sent by the first terminal, and recording the data reading timestamp and data type identifier of the current streaming data to be associated; If it is determined that the current streaming data to be associated fails to be associated with both the dimension data cache object and the first dimension table database, and the current streaming data to be associated meets a preset waiting and retrying condition, then determining a retry waiting time after the current streaming data to be associated enters a waiting and retrying mode; If the retry waiting time reaches the current latest waiting time, re-execute the association operation between the current streaming data to be associated and the first dimension table database; If the current streaming data to be associated is successfully associated with the first dimension table database, the first target dimension data associated with the current streaming data to be associated is added to the dimension data cache object, and the target streaming data is saved and output. The target streaming data consists of the current streaming data to be associated and its associated first target dimension data.
2. The method according to claim 1, It is characterized in that If the current streaming data to be associated fails to be associated with both the dimension data cache object and the first dimension table database, and the current streaming data to be associated meets a preset waiting and retrying condition, determining a retry waiting time after the current streaming data to be associated enters a waiting and retrying mode includes: Determine whether the current streaming data to be associated is successfully associated with the dimension data cache object; If it is determined that the current streaming data to be associated fails to be associated with the dimension data cache object, determining whether the current streaming data to be associated is successfully associated with the first dimension table database; If it is determined that the current streaming data to be associated fails to be associated with the first dimension table database, determining whether the current streaming data to be associated meets a preset waiting retry condition; If it is determined that the current streaming data to be associated meets the preset waiting and retry condition, then the retry waiting time after the current streaming data to be associated enters the waiting and retry mode is determined.
3. The method according to claim 2, It is characterized in that Determining whether the current streaming data to be associated meets a preset waiting retry condition includes: Determining whether the current streaming data to be associated is snapshot data based on the data type identifier; If the current streaming data to be associated is non-snapshot data, determining whether the number of waiting retries for the current streaming data to be associated to enter a waiting retry mode exceeds a preset number threshold; If the number of waiting retries does not exceed the preset number threshold, then calculating the time difference between the data reading timestamp of the current streaming data to be associated and the current system timestamp; Determine whether the time difference is less than or equal to a preset time difference threshold; If the time difference value is less than or equal to the preset time difference threshold, it is determined that the current streaming data to be associated meets the preset waiting retry condition.
4. The method according to any one of claims 1 to 3, It is characterized in that If the retry waiting time reaches the current latest waiting time, before re-executing the association operation between the current streaming data to be associated and the first dimension table database, the method further includes: Obtaining a minimum preset waiting time and a maximum preset waiting time for the current streaming data to be associated to enter the waiting and retrying mode; Counting the overall retry success rate of n historical streaming data to be associated; wherein the historical arrival time of the n historical streaming data to be associated arriving at the server is earlier than the current arrival time of the current streaming data to be associated arriving at the server, and n is an integer ≥ 0; The current latest waiting time is determined according to the minimum preset waiting time, the maximum preset waiting time and the overall retry success rate.
5. The method according to claim 4, It is characterized in that Determine the current latest waiting time according to the minimum preset waiting time, the maximum preset waiting time and the overall retry success rate, including: Determine a weight coefficient corresponding to the overall retry success rate; Obtain the last latest waiting time, and calculate the current required waiting time based on the last latest waiting time, the overall retry success rate and the weight coefficient; If the current required waiting time is greater than or equal to the minimum preset waiting time and less than or equal to the maximum preset waiting time, the current required waiting time is determined as the current latest waiting time; If the current required waiting time is less than the minimum preset waiting time, or greater than the maximum preset waiting time, the last latest waiting time is determined as the current latest waiting time.
6. The method according to claim 2, It is characterized in that Determining that the current streaming data to be associated fails to be associated with the first dimension table database includes: Check whether the connection between the first dimension table database and the connection pool is in a maintained connection state; If the connection between the first dimension table database and the connection pool is in a disconnected state, reestablishing the connection between the first dimension table database and the connection pool; When the connection between the first dimension table database and the connection pool is reestablished, obtaining the association field between the current streaming data to be associated and the first dimension table database; Querying whether there is first target dimension table data corresponding to the associated field in the first dimension table database; If the first target dimension table data corresponding to the associated field does not exist in the first dimension table database, it is determined that the current streaming data to be associated fails to be associated with the first dimension table database.
7. The method according to claim 1, It is characterized in that Determining the retry waiting time after the current streaming data to be associated enters the waiting retry mode includes: If it is determined that the waiting retry mode is the normal mode, the current streaming data to be associated is pushed from the original stream into the side output stream; The retry waiting time length of the current streaming data to be associated waiting for retry in the side output stream is calculated.
8. The method according to claim 1, It is characterized in that Listening and reading the current streaming data to be associated sent by the first terminal includes: Record the log location information at the current time point, and set a read lock for the streaming data table corresponding to the current streaming data to be associated; Reading the existing streaming data in the streaming data table in a snapshot form, and recording the data reading timestamp and data type identifier of the existing streaming data; After the reading is completed, the read lock of the streaming data table is released; Taking the log position information as the starting point, the changed streaming data of the streaming data table are read in sequence, and the data reading timestamp and data type identifier of the changed streaming data are recorded; wherein the current streaming data to be associated is the existing streaming data or the changed streaming data.
9. A device for delayed association of streaming data and dimensional data, It is characterized in that include: A creation module is configured to create a connection pool and a dimension data cache object corresponding to a first dimension table database, wherein first dimension data of the first dimension table database is synchronized with second dimension data of a second dimension table database of a business end; A reading module is configured to monitor and read the current streaming data to be associated sent by the first terminal, and record a data reading timestamp and a data type identifier of the current streaming data to be associated; The waiting module is configured to determine the retry waiting time after the current streaming data to be associated enters the waiting retry mode if it is determined that the current streaming data to be associated fails to be associated with the dimension data cache object and the first dimension table database, and the current streaming data to be associated meets the preset waiting retry condition; a retry module configured to re-execute the association operation between the current streaming data to be associated and the first dimension table database if the retry waiting time reaches the current latest waiting time; The output module is configured to add the first target dimension data associated with the current streaming data to be associated to the dimension data cache object if the current streaming data to be associated is successfully associated with the first dimension table database, save and output the target streaming data, and the target streaming data is composed of the current streaming data to be associated and its associated first target dimension data.
10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.
Citation Information
Patent Citations
Data correlation analysis method and system
CN108021696A
Method, system and device for realizing delay association of Flink about CDC data dimension table and storage medium
CN116910079A
Dimension table association optimization system, method, equipment and medium
CN117271507A
Techniques for verifiably processing data in distributed computing systems
US11386060B1
Joining and dimensional annotation in a streaming pipeline
US20210367994A1
