User position information association method, system and device based on Flink state calculation and medium

Abstract

The invention discloses a user position information association method, system and device based on Flink state calculation and a medium, belongs to the technical field of wireless communication, and aims to solve the technical problems of how to realize real-time association processing of large-scale massive signaling MME data and MRO data, reduce delay of data association result output and improve overall data processing efficiency. According to the technical scheme, an Flink and a Kafka cluster serve as basic software architecture, user signaling side data and MRO measurement data are collected to the Kafka cluster respectively, the Kafka cluster is consumed through the Flink, on the basis of Flink state calculation, real-time association of user signaling MME data and the MRO measurement data is achieved through the fast and efficient real-time calculation capacity of the Flink, and the user signaling MME data and the MRO measurement data are associated in real time. And backfilling the user mobile phone number information in the signaling side data into the MRO measurement data so as to realize real-time association of the user mobile phone number and the longitude and latitude position information.

Description

Technical Field

[0001] The present invention relates to the field of wireless communication technology, and in particular to a method, system, device and medium for associating user location information based on Flink state calculation. Background Art

[0002] The user signaling MME data contains the user's unique identifier and the user's mobile phone number information, and the MRO measurement data contains the user's unique identifier and longitude and latitude location information. In order to obtain the location information of the user's mobile phone number, the two types of data need to be associated according to the user's unique identifier and time to achieve the association between the user's mobile phone number and the location information. The traditional user mobile phone number and longitude and latitude location information data association technology is implemented based on the Hive offline data warehouse. Its disadvantages are low operating efficiency and low data timeliness, which cannot meet the scenarios with high real-time requirements.

[0003] Therefore, how to achieve real-time correlation processing of large-scale massive signaling MME data and MRO data, reduce the delay in outputting data correlation results, and improve the overall data processing efficiency is a technical problem that needs to be solved urgently. Summary of the invention

[0004] The technical task of the present invention is to provide a user location information association method, system, device and medium based on Flink state calculation to solve the problem of how to realize real-time association processing of large-scale massive signaling MME data and MRO data, reduce the delay in outputting data association results, and improve the overall data processing efficiency.

[0005] The technical task of the present invention is achieved in the following way: a user location information association method based on Flink state calculation, which uses Flink and Kafka cluster as the basic software architecture, collects user signaling side data and MRO measurement data to the Kafka cluster respectively, and consumes the Kafka cluster through Flink, based on Flink state calculation, and utilizes Flink's fast and efficient real-time computing capabilities to realize real-time association of user signaling MME data and MRO measurement data, and backfills the user mobile phone number information in the signaling side data into the MRO measurement data, thereby realizing real-time association of user mobile phone number and longitude and latitude location information.

[0006] As a preferred method, the user signaling side data and the MRO measurement data are collected to the Kafka cluster respectively as follows:

[0007] Parse user signaling MME data and MRO measurement data separately;

[0008] Write the parsing results to the topic corresponding to the Kafka cluster;

[0009] Set the number of partitions for the Kafka topic, use partitions to implement parallel data writing, give full play to Kafka's low latency and high throughput read and write advantages, and set the retention period of Kafka cached messages.

[0010] Better yet, use Flink's fast and efficient real-time computing capabilities to achieve real-time correlation between user signaling MME data and MRO measurement data as follows:

[0011] Use the receiver provided by Flink to read the real-time data stream of the Kafka cluster;

[0012] Use the connector provided by Flink to create a Kafka consumer to consume the user signaling MME data and MRO measurement data in the Kafka cluster.

[0013] Use Flink to create filters to filter out illegal data;

[0014] Use the converter provided by Flink to extract the core fields and convert them into a unified format. Extract the user's unique ID from the user signaling MME data and the MRO measurement data as the key. Use the user's unique ID as the association key to associate the real-time streams on both sides. Associate the mobile phone number information in the user signaling MME data with the longitude and latitude location information in the MRO measurement data. Set the Watermark size according to the latency requirements to avoid too much delay in triggering the association.

[0015] Enable Flink's CheckPoint mechanism and set the TTL size for retaining state data to avoid memory overflow caused by too large a state.

[0016] The association results are written to the OLAP database through a custom connector, and the association results are quickly queried and analyzed in multiple dimensions.

[0017] More optimally, setting a reasonable number of partitions for a Kafka Topic is to achieve highly parallel read and write and load balancing, and to achieve load balancing on the partitions to achieve maximum throughput;

[0018] The number of partitions is estimated based on the expected throughput and total data volume of the producers and consumers of each partition. The initial value of the partition is determined based on the amount of data received historically by any Topic, and then the throughput of the producers and consumers of the corresponding Topic is tested.

[0019] For example, the throughput of the producer and consumer of a Topic are Tp and Tc respectively, and the unit can be MB / s. The total target throughput is Tt, so the number of partitions = Tt / max(Tp, Tc).

[0020] Preferably, set the retention period of Kafka cached messages as follows:

[0021] The Kafka cluster provides multiple configuration items to manage message retention policies. The parameter of the configuration item is log.retention.hours. By adjusting the log.retention.hour configuration item, you can control the length of time Kafka retains messages.

[0022] A user location information association system based on Flink state calculation, the system is used to implement the above-mentioned user location information association method based on Flink state calculation; the system includes:

[0023] The data collection module is used to collect user signaling side data and MRO measurement data to the Kafka cluster respectively;

[0024] The correlation module is used to use Flink's fast and efficient real-time computing capabilities to achieve real-time correlation between user signaling MME data and MRO measurement data.

[0025] Preferably, the data acquisition module includes:

[0026] A parsing submodule, used to parse user signaling MME data and MRO measurement data respectively;

[0027] The writing submodule is used to write the parsing results to the topic corresponding to the Kafka cluster;

[0028] The partition setting submodule is used to set the number of partitions of Kafka Topic, use partitions to implement parallel data writing, give full play to Kafka's low latency and high throughput read and write advantages, and set the retention period of Kafka cache messages.

[0029] More preferably, the associated module includes:

[0030] The receiver provided by Flink is used to read the real-time data stream of the Kafka cluster;

[0031] The connector provided by Flink is used to create Kafka consumers to consume user signaling MME data and MRO measurement data in the Kafka cluster.

[0032] The filter provided by Flink is used to filter illegal data;

[0033] The converter provided by Flink is used to extract core fields and convert them into a unified format. The user's unique identifier is extracted from the user signaling MME data and the MRO measurement data as the key, and the real-time streams on both sides are associated with the user's unique identifier as the association key. The mobile phone number information in the user signaling MME data and the longitude and latitude location information in the MRO measurement data are associated together. The Watermark size is set according to the latency requirements to avoid too large a delay in triggering the association.

[0034] An electronic device comprising: a memory and at least one processor;

[0035] Wherein, the memory stores a computer program;

[0036] The at least one processor executes the computer program stored in the memory, so that the at least one processor executes the user location information association method based on Flink state calculation as described above.

[0037] A computer-readable storage medium stores a computer program, which can be executed by a processor to implement the user location information association method based on Flink state calculation as described above.

[0038] The user location information association method, system, device and medium based on Flink state calculation of the present invention have the following advantages:

[0039] (1) The present invention improves the real-time association of nearly 60T of user signaling MME data and MRO measurement data per day by mobile operators, improves the timeliness of data output, and ensures accurate one-time processing of data;

[0040] (ii) The present invention adopts the idea of ​​distributed parallel real-time computing and takes Flink's state computing as the basis to achieve accurate one-time association processing of user data and location information data, thereby improving association efficiency and reducing data output delay;

[0041] (III) From the perspective of distributed real-time processing, the present invention takes Flink's state computing capability as the basis, comprehensively considers the timeliness and consistency of data output, realizes real-time correlation processing of large-scale massive signaling MME data and MRO data, reduces the delay in outputting data correlation results, and improves the overall data processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The present invention is further described below in conjunction with the accompanying drawings.

[0043] Attached Figure 1 The figure is a flowchart of a method for associating user location information based on Flink state calculation. DETAILED DESCRIPTION

[0044] The user location information association method, system, device and medium based on Flink state calculation of the present invention are described in detail below with reference to the accompanying drawings and specific embodiments of the specification.

[0045] Embodiment 1:

[0046] As attached Figure 1 As shown, this embodiment provides a user location information association method based on Flink state calculation. The method uses Flink and Kafka cluster as the basic software architecture, collects user signaling side data and MRO measurement data to the Kafka cluster respectively, and consumes the Kafka cluster through Flink. Based on Flink state calculation, Flink's fast and efficient real-time computing capabilities are used to realize real-time association of user signaling MME data and MRO measurement data, and the user mobile phone number information in the signaling side data is backfilled into the MRO measurement data, thereby realizing real-time association of user mobile phone number and longitude and latitude location information.

[0047] In this embodiment, the user signaling side data and MRO measurement data are collected to the Kafka cluster respectively as follows:

[0048] ① Analyze user signaling MME data and MRO measurement data respectively;

[0049] ②Write the parsing results into the topic corresponding to the Kafka cluster;

[0050] ③ Set the number of partitions for the Kafka Topic, use partitions to implement parallel data writing, give full play to Kafka's low latency and high throughput read and write advantages, and set the retention period of Kafka cached messages.

[0051] In this embodiment, the fast and efficient real-time computing capability of Flink is used to realize the real-time association of user signaling MME data and MRO measurement data as follows:

[0052] ① Use the receiver provided by Flink to read the real-time data stream of the Kafka cluster;

[0053] ② Use the connector provided by Flink to create a Kafka consumer to consume the user signaling MME data and MRO measurement data in the Kafka cluster;

[0054] ③Use Flink to create filters to filter illegal data;

[0055] ④ Use the converter provided by Flink to extract the core fields and convert them into a unified format. Extract the user's unique identifier from the user signaling MME data and the MRO measurement data as the key, and use the user's unique identifier as the association key to associate the real-time streams on both sides. Associate the mobile phone number information in the user signaling MME data with the longitude and latitude location information in the MRO measurement data. Set the Watermark size according to the latency requirements to avoid too large a delay in triggering the association.

[0056] ⑤ Enable Flink's CheckPoint mechanism and set the TTL size for retaining state data to avoid memory overflow caused by excessive state;

[0057] ⑥ Write the association results to the OLAP database through a custom connector, and perform fast query and multi-dimensional analysis on the association results.

[0058] In this embodiment, setting a reasonable number of partitions of Kafka Topic is to achieve the purpose of highly parallel reading and writing and load balancing, and to achieve load balancing on the partitions to achieve maximum throughput;

[0059] The number of partitions is estimated based on the expected throughput and total data volume of the producers and consumers of each partition. The initial value of the partition is determined based on the amount of data received historically by any Topic, and then the throughput of the producers and consumers of the corresponding Topic is tested.

[0060] For example, the throughput of the producer and consumer of a Topic are Tp and Tc respectively, and the unit can be MB / s. The total target throughput is Tt, so the number of partitions = Tt / max(Tp, Tc).

[0061] In this embodiment, the retention period of Kafka cache messages is set as follows:

[0062] The Kafka cluster provides multiple configuration items to manage message retention policies. The parameter of the configuration item is log.retention.hours. By adjusting the log.retention.hour configuration item, you can control the length of time Kafka retains messages.

[0063] Embodiment 2:

[0064] This embodiment provides a user location information association system based on Flink state calculation, which is used to implement the user location information association method based on Flink state calculation in Example 1; the system includes:

[0065] The data collection module is used to collect user signaling side data and MRO measurement data to the Kafka cluster respectively;

[0066] The correlation module is used to use Flink's fast and efficient real-time computing capabilities to achieve real-time correlation between user signaling MME data and MRO measurement data.

[0067] The data acquisition module in this embodiment includes:

[0068] A parsing submodule, used to parse user signaling MME data and MRO measurement data respectively;

[0069] The writing submodule is used to write the parsing results to the topic corresponding to the Kafka cluster;

[0070] The partition setting submodule is used to set the number of partitions of Kafka Topic, use partitions to implement parallel data writing, give full play to Kafka's low latency and high throughput read and write advantages, and set the retention period of Kafka cache messages.

[0071] The association module in this embodiment includes:

[0072] The receiver provided by Flink is used to read the real-time data stream of the Kafka cluster;

[0073] The connector provided by Flink is used to create Kafka consumers to consume user signaling MME data and MRO measurement data in the Kafka cluster.

[0074] The filter provided by Flink is used to filter illegal data;

[0075] The converter provided by Flink is used to extract core fields and convert them into a unified format. The user's unique identifier is extracted from the user signaling MME data and the MRO measurement data as the key, and the real-time streams on both sides are associated with the user's unique identifier as the association key. The mobile phone number information in the user signaling MME data and the longitude and latitude location information in the MRO measurement data are associated together. The Watermark size is set according to the latency requirements to avoid too large a delay in triggering the association.

[0076] Embodiment 3:

[0077] This embodiment also provides an electronic device, including: a memory and a processor;

[0078] Wherein, the memory stores computer-executable instructions;

[0079] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the user location information association method based on Flink state calculation in any embodiment of the present invention.

[0080] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor may be a microprocessor or any conventional processor, etc.

[0081] The memory can be used to store computer programs and / or modules. The processor realizes various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function, etc.; the data storage area can store data created according to the use of the terminal, etc. In addition, the memory can also include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash memory card, at least one disk storage period, a flash memory device, or other volatile solid-state storage devices.

[0082] Embodiment 4:

[0083] This embodiment also provides a computer-readable storage medium, in which a plurality of instructions are stored, and the instructions are loaded by a processor, so that the processor executes the user location information association method based on Flink state calculation in any embodiment of the present invention. Specifically, a system or device equipped with a storage medium can be provided, on which a software program code that implements the functions of any of the above embodiments is stored, and a computer (or CPU or MPU) of the system or device reads and executes the program code stored in the storage medium.

[0084] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute a part of the present invention.

[0085] The storage medium embodiments for providing the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RYM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card, and a ROM. Alternatively, the program code can be downloaded from a server computer via a communication network.

[0086] In addition, it should be clear that the functions of any of the above embodiments can be implemented not only by executing the program code read by the computer, but also by enabling an operating system operating on the computer to complete part or all of the actual operations based on instructions from the program code.

[0087] In addition, it can be understood that the program code read from the storage medium is written to a memory provided in an expansion board inserted into the computer or written to a memory provided in an expansion unit connected to the computer, and then based on the instructions of the program code, a CPU installed on the expansion board or the expansion unit is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above-mentioned embodiments.

[0088] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A user location information association method based on Flink state calculation, characterized in that: This method uses Flink and Kafka cluster as the basic software architecture, collects user signaling side data and MRO measurement data to the Kafka cluster respectively, and consumes the Kafka cluster through Flink. Based on Flink state calculation, it uses Flink's fast and efficient real-time computing capabilities to realize real-time association of user signaling MME data and MRO measurement data, and backfills the user mobile phone number information in the signaling side data into the MRO measurement data, thereby realizing real-time association of user mobile phone number and longitude and latitude location information.

2. The user location information association method based on Flink state calculation according to claim 1 is characterized in that: The user signaling side data and MRO measurement data are collected to the Kafka cluster respectively as follows: Parse user signaling MME data and MRO measurement data separately; Write the parsing results to the topic corresponding to the Kafka cluster; Set the number of partitions for the Kafka topic, use partitions to implement parallel data writing, give full play to Kafka's low latency and high throughput read and write advantages, and set the retention period of Kafka cached messages.

3. The user location information association method based on Flink state calculation according to claim 1 or 2, characterized in that: The real-time association of user signaling MME data and MRO measurement data is achieved by using Flink's fast and efficient real-time computing capabilities as follows: Use the receiver provided by Flink to read the real-time data stream of the Kafka cluster; Use the connector provided by Flink to create a Kafka consumer to consume the user signaling MME data and MRO measurement data in the Kafka cluster. Use Flink to create filters to filter out illegal data; Use the converter provided by Flink to extract the core fields and convert them into a unified format. Extract the user's unique ID from the user signaling MME data and the MRO measurement data as the key. Use the user's unique ID as the association key to associate the real-time streams on both sides. Associate the mobile phone number information in the user signaling MME data with the longitude and latitude location information in the MRO measurement data. Set the Watermark size based on the latency requirements. Enable Flink's CheckPoint mechanism and set the TTL size for retaining state data; The association results are written to the OLAP database through a custom connector, and the association results are quickly queried and analyzed in multiple dimensions.

4. The user location information association method based on Flink state calculation according to claim 2 is characterized in that: Setting a reasonable number of partitions for Kafka Topic is to achieve highly parallel read and write and load balancing, and to achieve load balancing on partitions to achieve maximum throughput; The number of partitions is estimated based on the expected throughput and total data volume of the producers and consumers of each partition, and the initial value of the partition is determined based on the amount of data historically received by any Topic. The throughput of the producers and consumers of the corresponding Topic is then tested.

5. The user location information association method based on Flink state calculation according to claim 2 is characterized in that: The retention period for setting Kafka cache messages is as follows: The Kafka cluster provides multiple configuration items to manage message retention policies. The parameter of the configuration item is log.retention.hours. By adjusting the log.retention.hour configuration item, you can control the length of time Kafka retains messages.

6. A user location information association system based on Flink state calculation, characterized in that: The system is used to implement the user location information association method based on Flink state calculation according to any one of claims 1 to 5; the system includes: The data collection module is used to collect user signaling side data and MRO measurement data to the Kafka cluster respectively; The correlation module is used to use Flink's fast and efficient real-time computing capabilities to achieve real-time correlation between user signaling MME data and MRO measurement data.

7. The user location information association system based on Flink state calculation according to claim 6 is characterized in that: The data acquisition module includes: A parsing submodule, used to parse user signaling MME data and MRO measurement data respectively; The writing submodule is used to write the parsing results to the topic corresponding to the Kafka cluster; The partition setting submodule is used to set the number of partitions of Kafka Topic, use partitions to implement parallel data writing, give full play to Kafka's low latency and high throughput read and write advantages, and set the retention period of Kafka cache messages.

8. The user location information association system based on Flink state calculation according to claim 6 or 7, characterized in that: The associated modules include: The receiver provided by Flink is used to read the real-time data stream of the Kafka cluster; The connector provided by Flink is used to create Kafka consumers to consume user signaling MME data and MRO measurement data in the Kafka cluster. The filter provided by Flink is used to filter illegal data; The converter provided by Flink is used to extract core fields and convert them into a unified format. The user's unique identifier is extracted from the user signaling MME data and the MRO measurement data as the key, and the real-time streams on both sides are associated with the user's unique identifier as the association key. The mobile phone number information in the user signaling MME data and the longitude and latitude location information in the MRO measurement data are associated together, and the Watermark size is set according to the latency requirements.

9. An electronic device, characterized in that: include: memory and at least one processor; Wherein, the memory stores a computer program; The at least one processor executes the computer program stored in the memory, so that the at least one processor executes the user location information association method based on Flink state calculation according to any one of claims 1 to 5.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which can be executed by a processor to implement the user location information association method based on Flink state calculation according to any one of claims 1 to 5.

Images