Data processing methods, apparatus, equipment and computer-readable storage media

By caching file information in the base station server and dynamically allocating Flink acquisition resources, the problem of resource waste and computational pressure caused by inconsistent file sizes among base station servers is solved, thereby improving the efficiency and completeness of data acquisition.

CN116980960BActive Publication Date: 2026-11-17CHINA MOBILE GROUP ZHEJIANG +1
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202210455524.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-04-24
Publication Date
2026-11-17
Estimated Expiration
2042-04-24

AI Technical Summary

Technical Problem

In existing technologies, uneven allocation of Flink acquisition resources leads to resource waste and low data acquisition efficiency, especially when file data sizes are inconsistent between base station servers, causing computational pressure and data loss.

Method used

By obtaining the information of files to be downloaded from the base station server and caching it in Redis, the list of files to be downloaded and Flink collection resources are determined based on the file information. Collection resources are dynamically allocated to match the data size of each base station server, avoiding resource waste and computational pressure.

Benefits of technology

It achieves efficient allocation of Flink data collection resources, avoids waste of computing power and data loss, and ensures the uniqueness, integrity and transparency of data.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116980960B_ABST
    Figure CN116980960B_ABST
Patent Text Reader

Abstract

This invention discloses a data processing method, comprising: acquiring file information of files to be downloaded corresponding to multiple base station servers, and caching the file information in Redis, wherein the file information includes the data size of the files to be downloaded in each of the base station servers; determining a list of files to be downloaded corresponding to each base station server based on the file information in Redis; and determining collection resource information corresponding to each base station server based on the list of files to be downloaded and Flink collection resources. This invention also discloses a data processing apparatus, device, and computer-readable storage medium. This invention allocates Flink collection resources according to the data size of the files to be downloaded in each base station server, ensuring that when the Flink stream processing framework collects data from multiple base station servers, it avoids the problem of wasted computing power due to differences in file data size between different base station servers.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of data processing, and more particularly to a data processing method, apparatus, device, and computer-readable storage medium. Background Technology

[0002] With the widespread adoption and development of big data in the industry, the collection of basic data is becoming increasingly important. MR (Measurement Report) data, as crucial trajectory data within the O domain (operation support system), is of significant value in improving the accuracy of user location data through its collection, parsing, and analysis.

[0003] However, due to the large amount of MR data, which is stored in small files, and the severe skewness in MR file acquisition resources from the source due to the physical distribution of base stations, the MR data acquisition process includes: (1) regularly maintaining the base station server dimension table; (2) when the Flink program starts, forcing a balanced allocation of Flink acquisition resources for each base station server; (3) acquiring MR files according to the allocated Flink acquisition resources; (4) associating the acquired MR file data with the data in the base station server dimension table to obtain relatively complete data; and (5) parsing and storing the MR data files according to the parsing rules of different devices. Since the data size of the files to be acquired by each base station server is different, the balanced allocation of Flink acquisition resources leads to a waste of Flink acquisition resources, resulting in low efficiency in file data acquisition. Summary of the Invention

[0004] The main objective of this invention is to provide a data processing method, apparatus, device, and computer-readable storage medium, aiming to solve the technical problem of wasted Flink acquisition resources.

[0005] To achieve the above objectives, the present invention provides a data processing method, the data processing method comprising the following steps:

[0006] Obtain file information of files to be downloaded corresponding to multiple base station servers, and cache the file information in Redis, wherein the file information includes the data size of the files to be downloaded in each of the base station servers;

[0007] Based on the file information in Redis, a list of files to be downloaded corresponding to each of the base station servers is determined;

[0008] Based on the list of files to be downloaded and the Flink acquisition resources, the acquisition resource information corresponding to each base station server is determined.

[0009] Preferably, after the step of determining the collection resource information corresponding to each base station server based on the list of files to be downloaded and the Flink collection resources, the method further includes:

[0010] Based on the collected resource information, the base station server is subjected to data collection operations to obtain first data information;

[0011] The first data information is backfilled to obtain the second data information;

[0012] The second data information is stored in the database.

[0013] Preferably, the step of backfilling the first data information to obtain the second data information includes:

[0014] The first data information is parsed based on the base station information dimension table data to obtain different data tag information. Based on the data tag information, the first data information is concatenated to obtain the third data information.

[0015] Based on the last four digits of the user's mobile phone number in the third data information, the third data information is split to obtain the fourth data information;

[0016] The second data information is obtained by associating the fourth data information with the city and neighboring community fields.

[0017] Preferably, before the step of performing data collection on the base station server based on the collected resource information to obtain the first data information, the method further includes:

[0018] Based on the quota information and the data collection resource information, the target base station server for which a data collection connection needs to be established is determined, and a data collection connection is established with the target base station server.

[0019] Preferably, the step of determining the collection resource information corresponding to each base station server based on the list of files to be downloaded and the Flink collection resources includes:

[0020] Based on the list of files to be downloaded, determine the resource requirements for data collection corresponding to each of the base station servers;

[0021] Based on the resource requirements, the acquisition operators corresponding to each base station server in the Flink acquisition resources are determined, and the acquisition resource information is obtained.

[0022] Preferably, the step of obtaining file information of the files to be downloaded corresponding to multiple base station servers and caching the file information in Redis includes:

[0023] Obtain the reporting time of each file in each of the base station servers;

[0024] Based on the reported time, obtain the file information corresponding to each of the files to be downloaded;

[0025] The file information is cached in the Redis.

[0026] Preferably, the step of obtaining the file information corresponding to each of the files to be downloaded based on the reporting time includes:

[0027] Based on the reported time, determine the file to be downloaded corresponding to the base station server;

[0028] Obtain the file format corresponding to each of the files to be downloaded, the base station server identifier and base station identifier corresponding to the file to be downloaded, and the data size of the file to be downloaded;

[0029] The file information is determined based on the file format, the data size, the base station server identifier, and the base station identifier.

[0030] Furthermore, to achieve the above objectives, the present invention also provides a data processing apparatus, the data processing apparatus comprising:

[0031] The acquisition module is used to acquire file information of files to be downloaded corresponding to multiple base station servers and cache the file information in Redis, wherein the file information includes the data size of the files to be downloaded in each of the base station servers;

[0032] The first determining module is used to determine the list of files to be downloaded corresponding to each of the base station servers based on the file information in the Redis.

[0033] The second determining module is used to determine the collection resource information corresponding to each of the base station servers based on the list of files to be downloaded and the Flink collection resources.

[0034] In addition, to achieve the above objectives, the present invention also provides a data processing device, the data processing device comprising: a memory, a processor, and a data processing program stored in the memory and executable on the processor, wherein the data processing program, when executed by the processor, implements the steps of the data processing method as described above.

[0035] In addition, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing a data processing program, which, when executed by a processor, implements the steps of the data processing method described above.

[0036] This invention obtains file information of files to be downloaded from multiple base station servers and caches this information in Redis. The file information includes the data size of the files to be downloaded on each base station server. Based on the file information in Redis, a list of files to be downloaded corresponding to each base station server is determined. Based on the list of files to be downloaded and Flink acquisition resources, acquisition resource information corresponding to each base station server is determined. By allocating Flink acquisition resources according to the data size of the files to be downloaded on each base station server, this invention ensures that when the Flink stream processing framework collects data from multiple base station servers, it avoids the problem of wasted computing power due to differences in file data size between different base station servers, and the problem of data loss due to the computational pressure caused by uneven allocation of Flink acquisition resources, thus guaranteeing data uniqueness and integrity. Simultaneously, the process of caching the file information in Redis and then reading it ensures that subsequent data acquisition processes are not interfered with by the process of obtaining the file information of each file to be downloaded, improving data transparency. Attached Figure Description

[0037] Figure 1 This is a schematic diagram of the structure of a data processing device in the hardware operating environment involved in the embodiments of the present invention;

[0038] Figure 2 This is a flowchart illustrating the first embodiment of the data processing method of the present invention;

[0039] Figure 3 This is a schematic diagram of the functional modules of an embodiment of the data processing device of the present invention.

[0040] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0041] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0042] like Figure 1 As shown, Figure 1 This is a schematic diagram of the structure of a data processing device in the hardware operating environment involved in the embodiments of the present invention.

[0043] The data processing device in this embodiment of the invention can be a terminal device such as a computer that has data acquisition and processing functions.

[0044] like Figure 1As shown, the data processing device may include: a processor 1001, such as a CPU; a network interface 1004; a user interface 1003; a memory 1005; and a communication bus 1002. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen and an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or non-volatile memory, such as a disk drive. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0045] Optionally, the data processing device may also include a camera, RF (Radio Frequency) circuitry, sensors, audio circuitry, a WiFi module, and so on. Sensors such as light sensors, motion sensors, and other sensors will not be elaborated upon here.

[0046] Those skilled in the art will understand that Figure 1 The terminal structure shown does not constitute a limitation on the data processing device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0047] like Figure 1 As shown, the memory 1005, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a data processing program.

[0048] exist Figure 1 In the data processing device shown, the network interface 1004 is mainly used to connect to the backend server and communicate with the backend server; the user interface 1003 is mainly used to connect to the client (user terminal) and communicate with the client; and the processor 1001 can be used to call the data processing program stored in the memory 1005.

[0049] In this embodiment, the data processing device includes: a memory 1005, a processor 1001, and a data processing program stored in the memory 1005 and executable on the processor 1001. When the processor 1001 calls the data processing program stored in the memory 1005, it executes the steps of the data processing methods in the following embodiments.

[0050] The present invention also provides a data processing method, referring to Figure 2 , Figure 2This is a flowchart illustrating the first embodiment of the data processing method of the present invention.

[0051] In this embodiment, the data processing method includes the following steps:

[0052] Step S101: Obtain file information of the files to be downloaded corresponding to multiple base station servers, and cache the file information in Redis, wherein the file information includes the data size of the files to be downloaded in each of the base station servers;

[0053] In this embodiment, one base station server corresponds to one or more base stations. The base station is a public mobile communication base station, which is the interface device for mobile devices to access the Internet. MR data is the file data reported by each base station to the corresponding server under each base station server. The Flink stream processing framework can be used to collect resources from the base station server and obtain MR data.

[0054] Specifically, the process involves acquiring file information for files to be downloaded from multiple base station servers and caching this information in Redis. This file information refers to the file information of the files to be downloaded for data collection on each base station server. This information includes the data size and file format of each file. For example, a base station server scanning program periodically scans all equipment manufacturers' base station servers, primarily scanning each server to obtain base station information, file formats under each base station, and reporting times. This information is then stored for subsequent analysis. The base station information includes the number of base stations under each base station server and the rules for reporting files by each base station. The file path under the base station server is: base path / date / base station / file. The base station information under each server may change daily, and the filenames of the collected files will also change accordingly. Therefore, by analyzing the aforementioned base station information, file formats under each base station, and reporting times, the files to be downloaded on each base station server can be determined. After confirming the files to be downloaded, their file information, including the file size and file format, can be identified. This file information is then stored in Redis, an in-memory key-value database.

[0055] Step S102: Based on the file information in Redis, determine the list of files to be downloaded corresponding to each of the base station servers;

[0056] In this embodiment, the message queues of the aforementioned Redis are read and identified to determine the list of files to be downloaded on each server. This list comprises files for which data will be collected subsequently. Based on this list, the files to be downloaded on each server, along with their file information, including file size and format, can be identified. For each base station server, its corresponding list of files to be downloaded includes the filename (file identifier), file size, base station identifier of the base station to which the file belongs, and file format.

[0057] It should be noted that by using Redis as middleware, its high query speed can improve the efficiency of real-time program processing. At the same time, the process of obtaining the file information of the file to be downloaded corresponding to the base station server can be decoupled from the data collection process of the base station server in some other embodiments, avoiding mutual interference between the two processes and making the data obtained after data collection more transparent.

[0058] Step S103: Based on the list of files to be downloaded and the Flink acquisition resources, determine the acquisition resource information corresponding to each of the base station servers.

[0059] In this embodiment, based on the aforementioned list of files to be downloaded and the total amount of Flink acquisition resources, the allocation logic of Flink acquisition resources is determined, that is, the Flink acquisition resources allocated to each base station server. For example, when the Flink stream processing framework establishes connections to multiple base station servers for data acquisition, it can be assumed that the Flink acquisition resources include multiple acquisition operators. Based on the list of files to be downloaded, the data size of each file to be downloaded in each base station server is obtained. Then, based on the proportion of the data size of the files to be downloaded in each base station server, the proportion of acquisition operators allocated to each base station server in the Flink acquisition resources is determined, thereby determining the specific number of acquisition operators allocated to each base station server.

[0060] The data processing method proposed in this embodiment obtains file information of files to be downloaded corresponding to multiple base station servers and caches this file information in Redis. The file information includes the data size of the files to be downloaded on each base station server. Then, based on the file information in Redis, a list of files to be downloaded corresponding to each base station server is determined. Finally, based on the list of files to be downloaded and Flink acquisition resources, acquisition resource information corresponding to each base station server is determined. Allocating Flink acquisition resources according to the data size of the files to be downloaded on each base station server ensures that when the Flink stream processing framework collects data from multiple base station servers, it avoids the problem of wasted computing power due to differences in file data size between different base station servers, and the problem of data loss due to the computational pressure caused by uneven allocation of Flink acquisition resources, thus guaranteeing data uniqueness and integrity. At the same time, the process of caching file information in Redis and then reading it ensures that subsequent data acquisition processes are not interfered with by the process of obtaining the file information of each file to be downloaded, improving data transparency.

[0061] Based on the first embodiment, a second embodiment of the data processing method of the present invention is proposed. In this embodiment, the data processing method further includes:

[0062] Step S201: Based on the collected resource information, perform data collection operations on the base station server to obtain first data information;

[0063] Step S202: Perform a data backfilling operation on the first data information to obtain the second data information;

[0064] Step S203: Store the second data information in the database.

[0065] In this embodiment, based on the Flink acquisition resources allocated to each base station server, connections are established with each base station server and data is acquired. The acquired data information is the first data information. Data backfilling is performed on the obtained first data information, and then second data information is obtained and stored in the database. For example, the data backfilling operation for the first data information includes automatic dimension table loading, file parsing, data splitting, and data enhancement. First, the automatic dimension table loading process includes periodically retrieving data from HDFS (Hadoop Distributed File System). The system (Hadoop Distributed File System) automatically retrieves base station information dimension table data. The file parsing process includes: parsing the XML (Extensible Markup Language) in the first data information to obtain the tag information of each data, using the base station information dimension table data as a reference for the tag information, and concatenating the first data information by rows or columns according to the merging axis specified by the parameters in different tag information, that is, arranging and combining them in a fixed order, and then outputting the third data information; the data splitting process includes: splitting the third data information according to the last digit of the user's mobile phone number in the third data information, and outputting the fourth data information; the data enhancement process includes: associating the fourth data information with the city and neighboring cell fields in the fourth data information, outputting the second data information, and storing the second data information in the database.

[0066] In another embodiment, the data backfilling process may only include the automatic loading of dimension tables and file parsing. For example, the base station information dimension table data is automatically retrieved from HDFS (Hadoop Distributed File System) at regular intervals, the XML (Extensible Markup Language) in the first data information is parsed to obtain the tag information of each data, the base station information dimension table data is used as a reference for the tag information, and the first data information is row-concatenated or column-concatenated according to the merging axis specified by the parameters in different tag information, that is, arranged and combined in a fixed order, and then output as the second data information.

[0067] In another embodiment, the data backfilling process may only include the data splitting process. For example, the first data information is split according to the last three digits of the user's mobile phone number in the first data information, and the output is the second data information.

[0068] In another embodiment, the data backfilling process may only include the data enhancement process, for example, performing data association on the first data information based on the city and neighboring community fields in the first data information, and outputting it as the second data information.

[0069] It should be noted that, in the above embodiments, the data backfilling process includes at least one of the following three processes: automatic loading of dimension tables and file parsing, data splitting, and data enhancement. When two or more processes are included, the processes can be performed in any order, wherein the output data of the previous processing process is used as the input data of the next processing process.

[0070] Furthermore, in one embodiment, before step S201, the data processing method further includes:

[0071] Step S401: Based on the quota information and the collection resource information, determine the target base station server for which a data collection connection is to be established, and establish a data collection connection with the target base station server.

[0072] In this embodiment, the quota information includes the maximum number of base stations for each data collection session. Based on the quota information and collection resource information, the target base station servers for which data collection connections need to be established each time data collection is performed can be determined. Then, data collection connections are established with each target base station server. For example, the maximum number of base stations for each data collection session is determined, and then each target base station server is determined based on its corresponding collection resource information. This ensures that the total number of base stations corresponding to all target base station servers does not exceed the maximum number of base stations during each data collection session, and each target base station server receives its corresponding Flink collection resource allocation. By setting quotas to limit the number of base station servers for which data collection connections are established each time, the pressure caused by frequent connections to base station servers is reduced, data loss due to pressure is avoided, and the integrity and validity of the data are guaranteed.

[0073] The data processing method proposed in this embodiment involves collecting data from the base station server based on the collected resource information to obtain first data information; then performing a data backfilling operation on the first data information to obtain second data information; and finally storing the second data information in a database. This data backfilling operation makes the collected data information more complete and accurate, improving data integrity and usability.

[0074] Based on the second embodiment, a third embodiment of the data processing method of the present invention is proposed. In this embodiment, step S202 includes:

[0075] Step S301: Based on the base station information dimension table data, parse the first data information to obtain different data tag information, and perform data splicing on the first data information based on the data tag information to obtain the third data information;

[0076] Step S302: Based on the last four digits of the user's mobile phone number in the third data information, perform data splitting on the third data information to obtain the fourth data information;

[0077] Step S303: Based on the city and neighboring community fields in the fourth data information, perform data association on the fourth data information to obtain the second data information.

[0078] In this embodiment, the first data information is parsed based on the base station information dimension table data to obtain different data tag information. Then, the first data information is concatenated based on the data tag information to obtain the third data information. Subsequently, the third data information is split based on the last four digits of the user's mobile phone number to obtain the fourth data information. Finally, the fourth data information is associated based on the city and neighboring cell fields in the fourth data information to obtain the second data information. The base station information dimension table data is data obtained from HDFS periodically.

[0079] Specifically, the data backfilling process for the first data information follows a sequence of steps: dimension table autoloading, file parsing, data splitting, and data enhancement. For example, a parsing enhancement engine can be developed, comprising four logical blocks: dimension table autoloading, file parsing, data splitting, and data enhancement. The first data information is input into the parsing enhancement engine. The dimension table autoloading logic block periodically retrieves base station information dimension table data from HDFS. Then, the file parsing logic block parses the XML (Extensible Markup Language) in the first data information to obtain the tagging information for each data point. Using the base station information dimension table data as a reference for the tagging information, the data is then processed according to different tagging information. The parameters specifying the merging axis concatenate the first data information by row or column, that is, arrange and combine it in a fixed order to integrate the data. The concatenated data information is then output as the third data information to the data splitting logic block. Subsequently, the data splitting logic block splits the third data information according to the last digits of the user's mobile phone number. Based on the relevance of the last digits of the user's mobile phone number, the third data information is split into multiple data sets and output as the fourth data information to the data enhancement logic block. The fourth data information is then correlated according to the city and neighboring community fields. Finally, it is output as the second data information to the database.

[0080] The data processing method proposed in this embodiment parses the first data information based on base station information dimension table data to obtain different data marker information. Then, it concatenates the first data information based on the data marker information to obtain third data information. Next, it performs data splitting on the third data information based on the user's mobile phone number last two digits to obtain fourth data information. Finally, it performs data association on the fourth data information based on the city and neighboring cell fields to obtain the second data information. By backfilling the first data information, the collected data information becomes more complete, possessing higher data accuracy and practicality, thus improving data integrity and usability.

[0081] Based on the first embodiment, a fourth embodiment of the data processing method of the present invention is proposed. In this embodiment, step S103 includes:

[0082] Step S501: Based on the list of files to be downloaded, determine the resource requirements for data collection corresponding to each of the base station servers;

[0083] Step S502: Based on the resource demand, determine the acquisition operators corresponding to each base station server in the Flink acquisition resources, and obtain the acquisition resource information.

[0084] In this embodiment, based on the list of files to be downloaded corresponding to each base station server, the resource requirements of each base station server during data collection are determined. This resource requirement is the collection resources required by each base station server when Flink collects data from each base station server. Then, based on the above resource requirements and Flink collection resources, the collection resource information corresponding to each base station server is determined. For example, Flink collection resources include multiple collection operators, and the number of collection operators corresponds to the amount of Flink collection resources. Based on the list of files to be downloaded corresponding to each base station server, the data size of the files to be downloaded in each base station server is determined. Based on the proportion of the data size of the files to be downloaded in each base station server, the proportion of collection operators allocated to each base station server in Flink collection resources is determined. Then, based on the proportion, the number of collection operators allocated to each base station server is determined.

[0085] In some other embodiments, a preset value can be set based on the data size of the file to be downloaded. Based on the preset value, base station servers whose data size exceeds the preset value are designated as hot base station servers, and base station servers whose data size is less than the preset value are designated as unpopular base station servers. For example, when allocating Flink acquisition resources, the number of acquisition operators allocated to hot base station servers is twice the number of acquisition operators allocated to unpopular base station servers.

[0086] The data processing method proposed in this embodiment determines the resource requirements for data collection for each base station server based on the list of files to be downloaded. Then, based on the resource requirements, it determines the collection operators corresponding to each base station server in the Flink collection resources, thereby obtaining the collection resource information. When the Flink stream processing framework collects data from the base station servers, it allocates Flink collection resources according to the differences in the data size of the files to be downloaded on different base station servers. This avoids the waste of computing power caused by uneven allocation of Flink collection resources, as well as the data loss caused by excessive computing pressure due to uneven allocation of Flink collection resources, thus ensuring the validity and integrity of the data.

[0087] Based on the above embodiments, a fifth embodiment of the data processing method of the present invention is proposed. In this embodiment, step S101 includes:

[0088] Step S601: Obtain the reporting time of each file in each of the base station servers;

[0089] Step S602: Based on the reporting time, obtain the file information corresponding to each of the files to be downloaded;

[0090] Step S603: Cache the file information in the Redis.

[0091] Step S602 includes:

[0092] Step S701: Based on the reported time, determine the file to be downloaded corresponding to the base station server;

[0093] Step S702: Obtain the file format corresponding to each of the files to be downloaded, the base station server identifier and base station identifier corresponding to the file to be downloaded, and the data size of the file to be downloaded;

[0094] Step S703: Determine the file information based on the file format, the data size, the base station server identifier, and the base station identifier.

[0095] In this embodiment, each base station server is scanned to obtain the reporting time of each file on each base station server. Then, the reporting time of each file is analyzed to determine each file to be downloaded, and the file format, base station server identifier, and base station identifier corresponding to each file to be downloaded are obtained, as well as the data size of the file to be downloaded. This determines the file information of the file to be downloaded, and then the file information is cached in Redis. The file information includes the data size of each file to be downloaded, the base station server to which it belongs, the corresponding reporting base station, and the file format, etc. For example, a base station server scanner is used to periodically scan the base station servers of all equipment manufacturers. By scanning, the base station information of each base station server, the file format under each base station, and the reporting time of each file are obtained and stored for subsequent analysis. The base station server scanner is a program developed using the JAVA language. The base station information includes the number of base stations under each base station server and the rules for reporting files by each base station, etc. Then, based on the reporting time of each file, the files that need data collection reported by each base station after the previous scan are determined as files to be downloaded. The file format, base station server identifier, and base station identifier of each file to be downloaded are obtained, as well as the data size of each file to be downloaded. The data size of the file to be downloaded can be obtained directly. The file path under each base station server is: base path / date / base station / file. The base station information under each server may change every day, and the file name of the collected file will also change accordingly. Therefore, the file format reported by each base station at different time periods may be different. By analyzing the above base station information, the file format under each base station, and the reporting time of each file, and based on the file format, base station server identifier, and base station identifier of each file to be downloaded, the base station server to which each file to be downloaded belongs and the corresponding reporting base station can be determined. Thus, the file information of the file to be downloaded can be determined and cached in Redis.

[0096] In some other embodiments, when analyzing the aforementioned base station information, file formats under each base station, and reporting times of each file, a competing server operation can be performed. Specifically, the server information to be collected is pushed into a queue, and each parallel acquisition degree takes turns retrieving server information from the queue to connect. If the connection is successful, the connection information is saved; if it fails, it is put back into the queue for other parallel degrees to try connecting. This ensures that during subsequent data collection, each parallel degree collects data from only one server, and the same server is only collected once, avoiding duplicate data collection and missed data collection. In addition, base station files can be pre-generated. Based on the pattern of filenames to be collected under the server, filenames are generated in advance, instead of checking for newly added files each time, which facilitates subsequent data collection.

[0097] The data processing method proposed in this embodiment obtains the reporting time of each file in each base station server; then, based on the reporting time, obtains the file information corresponding to each file to be downloaded; and caches the file information in Redis. By periodically scanning and obtaining the reporting time of each file in each base station server, analyzing it, and obtaining the file information of each file to be downloaded and caching it in Redis, the dynamic acquisition of file information improves the timeliness and effectiveness of the data information obtained in subsequent data collection. At the same time, caching the file information in Redis avoids interference with subsequent data collection processes, improving the transparency of the data information obtained in subsequent data collection.

[0098] Furthermore, the present invention also provides a data processing apparatus, with reference to Figure 3 The data processing device includes:

[0099] The acquisition module 10 is used to acquire file information of files to be downloaded corresponding to multiple base station servers and cache the file information in Redis, wherein the file information includes the data size of the files to be downloaded in each of the base station servers;

[0100] The first determining module 20 is used to determine the list of files to be downloaded corresponding to each of the base station servers based on the file information in the Redis.

[0101] The second determining module 30 is used to determine the collection resource information corresponding to each of the base station servers based on the list of files to be downloaded and the Flink collection resources.

[0102] Furthermore, the video insertion device is also used for:

[0103] Based on the collected resource information, the base station server is subjected to data collection operations to obtain first data information;

[0104] The first data information is backfilled to obtain the second data information;

[0105] The second data information is stored in the database.

[0106] Furthermore, the video insertion device further includes: parsing the first data information based on base station information dimension table data to obtain different data tag information, and splicing the first data information based on the data tag information to obtain third data information;

[0107] Based on the last four digits of the user's mobile phone number in the third data information, the third data information is split to obtain the fourth data information;

[0108] The second data information is obtained by associating the fourth data information with the city and neighboring community fields.

[0109] Furthermore, the video insertion device also includes:

[0110] Based on the quota information and the data collection resource information, the target base station server for which a data collection connection needs to be established is determined, and a data collection connection is established with the target base station server.

[0111] Furthermore, the video insertion device also includes:

[0112] Based on the list of files to be downloaded, determine the resource requirements for data collection corresponding to each of the base station servers;

[0113] Based on the resource requirements, the acquisition operators corresponding to each base station server in the Flink acquisition resources are determined, and the acquisition resource information is obtained.

[0114] Furthermore, the video insertion device also includes:

[0115] Obtain the reporting time of each file in each of the base station servers;

[0116] Based on the reported time, obtain the file information corresponding to each of the files to be downloaded;

[0117] The file information is cached in the Redis.

[0118] Furthermore, the video insertion device also includes:

[0119] Based on the reported time, determine the file to be downloaded corresponding to the base station server;

[0120] Obtain the file format corresponding to each of the files to be downloaded, the base station server identifier and base station identifier corresponding to the file to be downloaded, and the data size of the file to be downloaded;

[0121] The file information is determined based on the file format, the data size, the base station server identifier, and the base station identifier.

[0122] The methods executed by the above-mentioned program units can be referred to in various embodiments of the data processing method of the present invention, and will not be repeated here.

[0123] Furthermore, this invention also proposes a data processing device, which includes: a memory, a processor, and a data processing program stored in the memory and executable on the processor. When the data processing program is executed by the processor, it implements the steps of the data processing method described above.

[0124] Furthermore, embodiments of the present invention also propose a computer-readable storage medium storing a data processing program, which, when executed by a processor, implements the steps of the data processing method described above.

[0125] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0126] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0127] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0128] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A data processing method, characterized in that, The data processing method includes the following steps: Obtain file information of files to be downloaded corresponding to multiple base station servers, and cache the file information in Redis, wherein the file information includes the data size of the files to be downloaded in each of the base station servers; Based on the file information in Redis, a list of files to be downloaded corresponding to each of the base station servers is determined; Based on the list of files to be downloaded and the Flink acquisition resources, the acquisition resource information corresponding to each base station server is determined. Specifically, the data size of the files to be downloaded in each base station server is determined according to the list of files to be downloaded. The proportion of acquisition operators allocated to each base station server in the Flink acquisition resources is determined according to the proportion of the data size of the files to be downloaded in each base station server. The number of acquisition operators allocated to each base station server is determined according to the proportion of acquisition operators. The step of determining the collection resource information corresponding to each base station server based on the list of files to be downloaded and the Flink collection resources further includes: Based on the collected resource information, the base station server is subjected to data collection operations to obtain first data information; The first data information is backfilled to obtain the second data information. The data backfilling operation includes at least one of the following: automatic dimension table loading, file parsing, data splitting, and data enhancement. Store the second data information in the database; The step of backfilling the first data information to obtain the second data information includes: Based on the base station information dimension table data, the Extensible Markup Language in the first data information is parsed to obtain different data markup information. Based on the merging axis specified by the parameters in the data markup information, the first data information is concatenated to obtain the third data information. The base station information dimension table data is data obtained periodically from the distributed file system HDFS. Based on the relevance of the user's mobile phone last digit in the third data information, the third data information is split into multiple datasets to form the fourth data information. The second data information is obtained by associating the fourth data information with the city and neighboring community fields.

2. The data processing method as described in claim 1, characterized in that, Before the step of collecting data from the base station server based on the collected resource information to obtain the first data information, the method further includes: Based on the quota information and the data collection resource information, the target base station server for which a data collection connection needs to be established is determined, and a data collection connection is established with the target base station server.

3. The data processing method as described in claim 1, characterized in that, The step of determining the collection resource information corresponding to each base station server based on the list of files to be downloaded and the Flink collection resources includes: Based on the list of files to be downloaded, determine the resource requirements for data collection corresponding to each of the base station servers; Based on the resource requirements, the acquisition operators corresponding to each base station server in the Flink acquisition resources are determined, and the acquisition resource information is obtained.

4. The data processing method according to any one of claims 1 to 3, characterized in that, The step of obtaining file information of the files to be downloaded corresponding to multiple base station servers and caching the file information in Redis includes: Obtain the reporting time of each file in each of the base station servers; Based on the reported time, obtain the file information corresponding to each of the files to be downloaded; The file information is cached in the Redis.

5. The data processing method as described in claim 4, characterized in that, The step of obtaining file information corresponding to each of the files to be downloaded based on the reporting time includes: Based on the reported time, the file to be downloaded corresponding to the base station server is determined; Obtain the file format corresponding to each of the files to be downloaded, the base station server identifier and base station identifier corresponding to the file to be downloaded, and the data size of the file to be downloaded; The file information is determined based on the file format, the data size, the base station server identifier, and the base station identifier.

6. A data processing apparatus, characterized in that, The data processing device includes: The acquisition module is used to acquire file information of files to be downloaded corresponding to multiple base station servers and cache the file information in Redis, wherein the file information includes the data size of the files to be downloaded in each of the base station servers; The first determining module is used to determine the list of files to be downloaded corresponding to each of the base station servers based on the file information in the Redis. The second determining module is used to determine the acquisition resource information corresponding to each base station server based on the list of files to be downloaded and the Flink acquisition resources. Specifically, the data size of the files to be downloaded in each base station server is determined according to the list of files to be downloaded, the proportion of acquisition operators allocated to each base station server in the Flink acquisition resources is determined according to the proportion of the data size of the files to be downloaded in each base station server, and the number of acquisition operators allocated to each base station server is determined according to the proportion of acquisition operators. The second determining module is further configured to perform data collection operations on the base station server based on the collected resource information to obtain first data information; perform data backfilling operations on the first data information to obtain second data information, wherein the data backfilling operations include at least one of: dimension table autoloading, file parsing, data splitting, and data enhancement; and store the second data information in the database. The second determining module is further configured to parse the Extensible Markup Language (XML) in the first data information based on the base station information dimension table data to obtain different data marker information; to concatenate the first data information based on the merging axis specified by the parameters in the data marker information to obtain third data information, wherein the base station information dimension table data is data obtained periodically from the distributed file system HDFS; to perform data splitting on the third data information based on the relevance of the user's mobile phone number last digits in the third data information to obtain multiple datasets combined into fourth data information; and to perform data association on the fourth data information based on the city and neighboring cell fields in the fourth data information to obtain the second data information.

7. A data processing device, characterized in that, The data processing device includes: a memory, a processor, and a data processing program stored in the memory and executable on the processor, wherein the data processing program, when executed by the processor, implements the steps of the data processing method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The readable storage medium stores a data processing program, which, when executed by a processor, implements the steps of the data processing method as described in any one of claims 1 to 5.

Citation Information

Patent Citations

  • Resource collection method, resource collection device, storage medium and electronic equipment

    CN114003566A

  • Stream processing method and system for unstructured file based on distributed architecture

    CN114328435A