Data processing method, device and electronic equipment

Abstract

The embodiment of the present invention discloses a data processing method, device and electronic device, the method comprising: based on a preset data interface, collecting target data for a target service, the target data comprising signaling data and / or user plane data for the target service; placing the target data into a preset lock-free queue, and classifying the target data based on preset classification rules and data features of the target data to obtain multiple classified data sets; based on the lock-free queue, obtaining a first data set that meets preset computing requirements and a second data set that meets preset loading requirements from the classified data sets; based on the lock-free queue, distributing the target data in the first data set to a preset message publishing system, and loading the target data in the second data set to a preset distributed file system. Through this method, the data processing efficiency during data distribution and loading can be improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data processing method, device and electronic equipment. Background Art

[0002] With the continuous development of computer technology, the number of users of mobile communication services has increased dramatically. How to meet the real-time computing needs of mobile communication services with large data volumes (such as real-time computing, offline computing, etc.) has become the focus of operators.

[0003] At present, multiple data streams can be distributed through the linked list-based double-ended blocking queue of the log server, that is, the mobile communication business data can be pushed to the message publishing system (such as Kafka) through the log server, and at the same time, the mobile communication business data can also be loaded into the distributed file system (Hadoop Distributed FileSystem, HDFS) through the log server.

[0004] However, since the linked list-based double-ended blocking queue implements data entry and exit based on an exclusive lock, the distribution and loading of mobile communication data in the above manner will result in delayed execution due to threads waiting for unlocking. In addition, since the amount of mobile communication data to be processed is large, and the data distribution method of the log server has a slow distribution speed, the timeliness of data processing is poor and the data processing efficiency is low. Summary of the invention

[0005] The purpose of the embodiments of the present invention is to provide a data processing method, device and electronic device to solve the problem of low data processing efficiency when distributing and loading data in the prior art.

[0006] To solve the above technical problems, the embodiments of the present invention are implemented as follows:

[0007] In a first aspect, an embodiment of the present invention provides a data processing method, the method comprising, based on a preset data interface, collecting target data for a target service, the target data comprising signaling data and / or user plane data for the target service; placing the target data into a preset lock-free queue, and classifying the target data based on preset classification rules and data features of the target data to obtain multiple classified data sets, the lock-free queue being a queue constructed based on a circular array and not requiring locking; based on the lock-free queue; based on the lock-free queue, obtaining a first data set in the classified data set that meets preset computing requirements and a second data set that meets preset loading requirements, distributing the target data in the first data set to a preset message publishing system, and loading the target data in the second data set to a preset distributed file system.

[0008] In a second aspect, an embodiment of the present invention provides a data processing device, comprising: a data acquisition module, for collecting target data for a target service based on a preset data interface, the target data comprising signaling data and / or user plane data for the target service; a data placement module, for placing the target data into a preset lock-free queue, and classifying the target data based on preset classification rules and data features of the target data to obtain multiple classified data sets, the lock-free queue being a queue constructed based on a circular array and not requiring locking; a data set acquisition module, for acquiring, based on the lock-free queue, a first data set in the classified data set that meets preset computing requirements and a second data set that meets preset loading requirements; a data processing module, for distributing the target data in the first data set to a preset message publishing system based on the lock-free queue, and loading the target data in the second data set to a preset distributed file system.

[0009] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the data processing method provided in the above embodiment.

[0010] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the data processing method provided in the above embodiment are implemented.

[0011] It can be seen from the technical solution provided by the above embodiment of the present invention that the embodiment of the present invention collects target data for the target service based on the preset data interface, the target data includes signaling data and / or user plane data for the target service, puts the target data into the preset lock-free queue, and classifies the target data based on the preset classification rules and the data characteristics of the target data to obtain multiple classified data sets, and obtains the first data set that meets the preset computing requirements and the second data set that meets the preset loading requirements in the classified data set based on the lock-free queue. The lock-free queue is a queue built based on a ring array and does not need to be locked. Based on the lock-free queue, the target data in the first data set is distributed to the preset message publishing system, and the target data in the second data set is loaded into the preset distributed file system. In this way, through the architecture design of the lock-free queue, the lock-free and non-blocking characteristics of the ring array can be fully utilized, and the target data in the first data set that meets the preset computing requirements in the collected target data is distributed to the preset message publishing system, and the target data in the second data set that meets the preset loading requirements is loaded into the preset distributed file system, which can improve the collection, processing, distribution and loading efficiency of the target data, reduce data latency, reduce resource consumption, and improve the data processing efficiency when distributing and loading data. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0013] Figure 1 A schematic diagram of a data processing method of the present invention;

[0014] Figure 2 A schematic flow chart of another data processing method of the present invention;

[0015] Figure 3 It is a structural schematic diagram of a data processing device of the present invention;

[0016] Figure 4 The figure is a schematic structural diagram of an electronic device of the present invention. DETAILED DESCRIPTION

[0017] Embodiments of the present invention provide a data processing method, device and electronic device.

[0018] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0019] Embodiment 1

[0020] like Figure 1 As shown, an embodiment of the present invention provides a data processing method, the execution subject of the method may be a server, the server may be an independent server, or a server cluster composed of multiple servers. The method may specifically include the following steps:

[0021] In S102, target data for the target business is collected based on a preset data interface.

[0022] Among them, the target service can be any mobile communication service, the target data can include signaling data and / or user plane data for the target service, and the preset data interface can be any data interface that can obtain the target data. For example, the preset data interface can include a Kafka data interface, a preset File Transfer Protocol (FTP) interface, a WebService interface, etc.

[0023] Among them, the terminal electronic device can be any electronic device held by the on-site staff and used to collect the base station parameters of the target base station. For example, the terminal electronic device can be any mobile terminal electronic device such as a mobile phone or a tablet computer. The target base station can be any base station to be opened for providing mobile communication services to users. The base station parameters can be parameters used to describe the basic situation of the target base station. For example, the base station parameters can be parameters such as the board serial number, board model, location information of the target base station, and the network standard of the target base station.

[0024] In practice, with the continuous development of computer technology, the number of users of mobile communication services has increased dramatically. How to meet the real-time computing needs of mobile communication services with large data volumes (such as real-time computing, offline computing, etc.) has become the focus of operators.

[0025] At present, the log server can distribute multiple data streams through a linked list-based double-ended blocking queue, that is, the mobile communication service data is pushed to the message publishing system (such as Kafka) through the log server, and the mobile communication service data can also be loaded into the distributed file system (Hadoop Distributed FileSystem, HDFS) through the log server at the same time. However, since the linked list-based double-ended blocking queue implements data entry and exit based on an exclusive lock, the distribution and loading of mobile communication data in the above manner will cause delayed execution due to threads waiting to release the lock. In addition, since the amount of mobile communication data to be processed is large, and the data distribution method of the log server has a slow distribution speed, the timeliness of data processing is poor and the data processing efficiency is low.

[0026] In addition, mobile communication data can be collected through the log server and pushed to the message publishing system, and then the subsequent landing program can load the mobile communication data into the distributed file system. However, since the above method requires the message publishing system to forward the mobile communication data and the subsequent landing site to load the mobile communication data, there are many operation links in the distribution and loading process, and the data processing efficiency is low. In addition, there is also a single point problem of the data link in the entire data processing process, that is, if the message publishing system fails, the entire data link cannot operate, and the system stability is poor.

[0027] To this end, the embodiment of the present invention provides another implementation solution, which may specifically include the following contents:

[0028] Different types of target data for target businesses can be obtained through different preset data interfaces. For example, Kafka data for target businesses can be obtained through the Kafka interface, and files on the FTP server can be scanned in a scheduled manner through the FTP interface to obtain corresponding target data. Interface data (i.e., target data) can be obtained through the WebService interface by establishing a service interface.

[0029] In actual application scenarios, there may be a variety of different preset data interfaces, which are not specifically limited in the embodiments of the present invention.

[0030] In S104, the target data is put into a preset lock-free queue, and the target data is classified based on preset classification rules and data features of the target data to obtain a plurality of classified data sets.

[0031] The lock-free queue may be a queue constructed based on a circular array and does not need to be locked. In addition, the lock-free queue may be a queue capable of controlling the queue capacity, that is, the lock-free queue may be a lock-free bounded queue.

[0032] In the implementation, the lock-free queue is taken as Disruptor as an example, that is, the lock-free queue can adopt the producer and consumer mode. It is a lock-free queue constructed by the Ring Buffer ring array, and the data in the lock-free queue can be stored and updated. The position of the element (that is, the data in the lock-free queue) in the lock-free queue can be quickly located through bit operations. The Disruptor queue can define the Sequence interface (that is, the sequentially increasing sequence number) to number and manage the data exchanged through the queue, and the data processing process can be processed one by one along the sequence number. The core of the Disruptor queue is the Sequencer interface, which has two implementation classes, namely SingleProducerSequencer and MultiProducerSequencer. These two implementation classes can define concurrent algorithms for quickly and correctly transmitting data between producers and consumers. The preset waiting strategy (i.e. Wait Strategy) determines how a consumer waits for the producer to put data (or events) into the Disruptor queue. The data put into the Disruptor queue can be the data passed from the producer to the consumer. The EventProcessor in the Disruptor queue can be used to process the data in the Disruptor queue. The Consumer needs to implement the EventHandler interface to perform data consumption processing. The Producer can publish the logic code of the event for the Disruptor queue.

[0033] The target data can be put into the above-mentioned Disruptor as an example, and then the target data can be classified based on the preset classification rules and the data characteristics of the target data to obtain multiple classified data sets, wherein the preset classification rules can be classification rules preset according to the business attributes of the target business, and the data characteristics of the target data can include data characteristics such as the data type and data size of the target data.

[0034] In S106 , based on the lock-free queue, a first data set that meets a preset computing requirement and a second data set that meets a preset loading requirement are obtained from the classified data sets.

[0035] The preset computing requirement may be a computing requirement determined according to a real-time computing task, and the preset loading requirement may be a computing requirement determined according to an offline computing task.

[0036] In implementation, the classified data sets contained in the lock-free queue may be searched according to the classification identifier to obtain a first data set that meets a preset computing requirement and a second data set that meets a preset loading requirement.

[0037] In S108, based on the lock-free queue, the target data in the first data set is distributed to a preset message publishing system, and the target data in the second data set is loaded into a preset distributed file system.

[0038] In implementation, the target data in the second data set can be written into a data cache queue, and then based on a preset cache cycle, the target data in the second data set stored in the data cache queue can be cached in a disk, and then the cached data in the disk can be processed on the spot to load the target data in the second data set after the processing on the spot into a preset distributed file system.

[0039] The embodiment of the present invention provides a data processing method, based on a preset data interface, to collect target data for a target service, the target data includes signaling data and / or user plane data for the target service, put the target data into a preset lock-free queue, and classify the target data based on preset classification rules and data characteristics of the target data to obtain multiple classified data sets, based on the lock-free queue, obtain a first data set that meets the preset computing requirements and a second data set that meets the preset loading requirements in the classified data sets, the lock-free queue is a queue built based on a ring array and does not need to be locked, based on the lock-free queue, distribute the target data in the first data set to a preset message publishing system, and load the target data in the second data set to a preset distributed file system. In this way, through the architecture design of the lock-free queue, the lock-free and non-blocking characteristics of the ring array can be fully utilized, the target data in the first data set that meets the preset computing requirements in the collected target data can be distributed to the preset message publishing system, and the target data in the second data set that meets the preset loading requirements can be loaded into the preset distributed file system, which can improve the collection, processing, distribution and loading efficiency of the target data, reduce data latency, reduce resource consumption, and improve the data processing efficiency when distributing and loading data.

[0040] Embodiment 2

[0041] like Figure 2 As shown, an embodiment of the present invention provides a data processing method, the execution subject of the method may be a server, the server may be an independent server, or a server cluster composed of multiple servers. The method may specifically include the following steps:

[0042] In S202, the queue capacity of the lock-free queue is obtained, and based on the queue capacity, target data for the target business is collected through a preset data interface.

[0043] In implementation, by collecting target data through the queue capacity of the lock-free queue, data interaction time can be reduced, and data waiting problems caused by the amount of target data being greater than the queue capacity of the lock-free queue can be avoided.

[0044] A data collection thread can be defined to collect target data for a target business through a preset data interface.

[0045] The lock-free queue can be a public static variable, and the production mode can be defined as ProducerType.MULTI, that is, multi-producer mode.

[0046] In S204, the queue capacity of the lock-free queue and data information of the target data are obtained.

[0047] The data information of the target data may include information such as the data volume and data type of the target data.

[0048] In S206 , a data placement speed is determined based on the queue capacity of the lock-free queue and the data information of the target data, and the target data is placed in a preset lock-free queue based on the data placement speed.

[0049] In implementation, the data insertion speed can be determined based on the data volume of the target data and the queue capacity of the lock-free queue, or the data insertion order and the corresponding insertion speed can be determined based on the data volume of target data of different data types and the queue capacity of the lock-free queue.

[0050] In actual application scenarios, there may be various methods for determining the data placement speed of the target data into the preset lock-free queue, which may vary according to different actual application scenarios, and the embodiments of the present invention do not specifically limit this.

[0051] By determining the insertion speed, the data inflow speed of the lock-free queue can be coordinated to avoid problems such as memory overflow.

[0052] A circular array object may be defined to produce data based on the target data through the circular data object after the data acquisition thread acquires the target data.

[0053] In S208, data information of the target data is obtained.

[0054] In S210, a preset cleaning rule is determined based on the data information.

[0055] In implementation, for example, the number of cleaning objects for cleaning can be determined according to the data volume of the target data to meet the needs of actual data calculation and improve the cleaning efficiency.

[0056] In S212, based on the preset cleaning rules, the target data in the lock-free queue is cleaned, and based on the preset classification rules and the data features of the cleaned target data, the cleaned target data is classified to obtain multiple classified data sets.

[0057] In implementation, a classification object pool may be defined to classify the cleaned target data based on preset classification rules and data features of the cleaned target data through the classification object pool to obtain multiple classification data sets.

[0058] When the target data in the lock-free queue is cleaned based on the preset cleaning rules, the cleaning conversion operation group can be initialized, and GuavaCache can be used to update the preset clear rules regularly. In this way, the network IO interaction with the database can be reduced, and the hot switching of rule data can be realized. By updating the preset clear rules, the dynamic parsing ability of real-time distribution and loading of the lock-free queue can be improved. And according to the data volume of the target data actually collected, the number of cleaning conversion objects can be set to meet the actual data calculation requirements and achieve the purpose of high-performance cleaning conversion.

[0059] The data source identifier can be used to define the parsing method of the data field, parse the target data and perform data cleaning processing. The Aviator rule engine expression can be used to define data cleaning rules to enhance the data cleaning capability of the device. For example, the description of some fields can be shown in Table 1 below.

[0060] Table 1

[0061]

[0062] It is also possible to define fields related to data source identification to facilitate data content field parsing, and to use Aviator to reorganize fields to generate new field patterns, achieve high-performance real-time data expansion, and enhance the data expansion capabilities of the device. For example, the description of some fields can be shown in Table 2 below.

[0063] Table 2

[0064]

[0065] You can make full use of GuavaCache's data caching capabilities to construct cleaning rules and enhance configurability, so as to achieve quasi-real-time cleaning rule updates and hot-load cleaning rule effectiveness.

[0066] The receiving objects of each link of the lock-free queue can be defined, and the data source identifier is used as the identification identifier of the data packet to facilitate the acquisition of classification information, and the original message content is stored in the form of a byte array. The converted data objects can be stored through the HashMap model to facilitate Aviator rule operations. The list model can be defined to store classification objects to achieve fast data distribution and loading. Function preloading can be implemented for Aviator to facilitate the use of cleaning and conversion operation objects, which can reduce the execution time of function loading. TransformHandler can be defined to implement the WorkHandler interface class of the lock-free queue, and high-speed data cleaning and conversion work can be performed in combination with the Aviator rule engine according to the rule definition.

[0067] When classifying, you can define the classification rules for the target data. You can use Aviator rules to define the classification rules, and use the rule calculation result True or False to determine whether it is this type of data, thereby improving the efficiency of data processing.

[0068] The classification calculation rules corresponding to each type of classification data set, as well as the location and field name information of its output, can be defined to enhance the data classification configurability of the device. The classification definition table can be shown in Table 3 below.

[0069] Table 3

[0070]

[0071] The classification field output table can be shown in Table 4 below.

[0072] Table 4

[0073] Field Name Field Description Classification Number Data classification number Output field name Data classification number Output field locations Identify output field locations

[0074] GuavaCache can be used to cache classification information configuration information to enhance hot switching of classification rules.

[0075] You can also define independent classification operation objects to increase classification rule operations and quickly implement data classification, so as to enhance the ability of data division processing and rapid classification and achieve high-performance data classification processing.

[0076] In S214, based on the lock-free queue, a first data set that meets the preset computing requirement and a second data set that meets the preset loading requirement are obtained from the classified data sets.

[0077] In S216, based on the lock-free queue, the target data in the first data set is distributed to a preset message publishing system.

[0078] In implementation, a file writing thread may be defined to push the target data in the first data set into the writing thread, and the file writing thread configures the writing file and performs a file writing operation, that is, distributing the target data in the first data set to a preset message publishing system.

[0079] In addition, a file writing management thread can be defined to monitor the operation of the file writing thread. In addition, the file writing management thread can also monitor the amount of unprocessed data of the target data written into the lock-free queue, collect data information of the target data sent to the lock-free queue, and notify the data collection thread to limit the collection speed.

[0080] In order to meet the performance requirements of real-time distribution to Kafka, an independent Kafka production operation group can be used to randomly obtain the Kafka production instance of the Kafka production instance group to push the target data in the first data set to the Kafka corresponding to the data. The configurable execution object data mode can be adopted to dynamically adjust the execution object according to the actual data volume of the target data in the first data set, increase the parallelism of the distribution, and achieve the effect of high-performance distribution.

[0081] You can define the distributed Kafka service parameters and define the data classification distribution rules. Some field descriptions can be shown in Table 5 below.

[0082] Table 5

[0083] Field Name Field Description Classification Number Data classification number Distribute Kafka corresponding IP Kafka server corresponding IP address Distribute Kafka corresponding topics Kafka server corresponding to the write topic

[0084] GuavaCache can be used to cache the distribution rules of corresponding categories to facilitate quick retrieval of rule objects, improve data reading performance, and reduce IO interactions between the database and the network.

[0085] According to the corresponding cache rules, you can take out the corresponding production instance from the production instance group, use the Kafka producer API to submit data to the corresponding Kafka server, and complete real-time data distribution.

[0086] In S218, based on the asynchronous loading method, the target data in the second data set is loaded into the preset distributed file system.

[0087] In implementation, the target data in the second data set can be loaded into the preset distributed file system based on the preset loading cycle and the preset loading amount. By loading the target data in the second data set into the preset distributed file system in a timed and quantitative manner, the data interaction process can be reduced and the efficiency of data processing can be improved.

[0088] Before collecting target data, you can initialize system parameters to load system configuration files for data processing. For example, you can initialize the number of objects in each operation group, the number of Kafka production instances, the number of collection threads, the number of file writing threads, the number of file loading threads, etc. Provide parameter configuration information for the use of lock-free queues, and build the number of producers and consumers of the queue based on the configuration information.

[0089] When initializing the Kafka production instance group, since the Kafka instance is thread-safe, multiple threads can share a Kafka instance. To prevent initializing the Kafka connection every time, a considerable number of Kafka instances can be preset to improve the throughput of Kafka production. The Kafka production instance pool can be used to provide Kafka producers for the use of lock-free queues to avoid connection instability and data loss caused by generating too many Kafka connections.

[0090] When classifying the target data, you can initialize the file writing thread group, use the MpscLinkedQueue lock-free queue to implement file operation processing, so as to enhance the throughput of data exchange between threads, and reduce disk IO interaction by batch and timed data writing to achieve high-performance file writing. To achieve the writing of target data in the lock-free queue, you can use the method of thread operation of independent files to prevent multiple threads from operating the same file or shared variables together. The following process can be used to achieve the writing of target data in the data lock-free queue.

[0091] First, you can define the classification writing rules, define the file name rules, file format, compression format, separator, line break, and the number of initialization files (i.e., the number of multi-file writing threads, in which different files can be written) for each classification data set, and each file (i.e., each classification data set) is operated by one thread to avoid file sharing between threads. For example, some classification writing rules can be shown in Table 6 below.

[0092] Table 6

[0093] Field Name Field Description Classification Number Data classification number File name rules Such as: xxx_yyyymmdd_xxxx.avl.... File Format Such as: txt,orcfile. Compression format For example: zip... is empty and does not compress Delimiter Such as:, ... applicable to txt Line Breaks Such as: / n....Applicable to txt Initialization file number The number of operation threads

[0094] Secondly, you can define the write field rules to specify the field name, write location, field type, and field length information to be written, so as to organize the data information, facilitate the rapid and reasonable generation of orc files, and quickly load them into the database. For example, some classification write rules can be shown in Table 7 below.

[0095] Table 7

[0096] Field Name Field Description Classification Number Data classification number Field Name The field name to be written Field location Field write position Field Type Preset field type, such as string int Field length Preset field length

[0097] Based on the above parameter definitions, a file writing implementation class can be constructed, and the interface design pattern can be used to implement data adaptation writing. The factory design pattern can define the association between the classification and the file writing entity class, and can quickly read the data writing object corresponding to the classification data set.

[0098] A data center can be configured in the server. The data center can be used to manage the resident thread for writing classified data sets. It can initialize the classified file writing thread, return the data operation thread according to the classification, and monitor the execution of the file writing data. The data center can coordinate the inflow and outflow rates of the target data, and can adopt a back pressure mechanism to prevent memory overflow problems.

[0099] In addition, you can also use the thread-guarding round-robin lock-free queue method to flush the target data to the disk in a timed and quantitative manner to reduce the data interaction time between threads and disk IO operations, thereby speeding up the file writing speed and improving data processing efficiency.

[0100] The file loading thread can be started according to the set parameters through the file loading thread group. After receiving the target data in the second data set that has been written, the file loading thread can load the received data into HDFS through the Hadoop SDK and mount the partition to HIVE at a fixed time. After the loading is completed, HIVE can immediately query the loaded data.

[0101] The embodiment of the present invention provides a data processing method, based on a preset data interface, to collect target data for a target service, the target data includes signaling data and / or user plane data for the target service, put the target data into a preset lock-free queue, and classify the target data based on preset classification rules and data characteristics of the target data to obtain multiple classified data sets, based on the lock-free queue, obtain a first data set that meets the preset computing requirements and a second data set that meets the preset loading requirements in the classified data sets, the lock-free queue is a queue built based on a ring array and does not need to be locked, based on the lock-free queue, distribute the target data in the first data set to a preset message publishing system, and load the target data in the second data set to a preset distributed file system. In this way, through the architecture design of the lock-free queue, the lock-free and non-blocking characteristics of the ring array can be fully utilized, the target data in the first data set that meets the preset computing requirements in the collected target data can be distributed to the preset message publishing system, and the target data in the second data set that meets the preset loading requirements can be loaded into the preset distributed file system, which can improve the collection, processing, distribution and loading efficiency of the target data, reduce data latency, reduce resource consumption, and improve the data processing efficiency when distributing and loading data.

[0102] Embodiment 3

[0103] The above is a data processing method provided by an embodiment of the present invention. Based on the same idea, an embodiment of the present invention also provides a data processing device, such as Figure 3 shown.

[0104] The data processing device comprises: a data acquisition module 301, a data input module 302, a data set acquisition module 303 and a data processing module 304, wherein:

[0105] The data collection module 301 is used to collect target data for a target service based on a preset data interface, where the target data includes signaling data and / or user plane data for the target service;

[0106] The data putting module 302 is used to put the target data into a preset lock-free queue, and classify the target data based on preset classification rules and data features of the target data to obtain multiple classified data sets. The lock-free queue is a queue built based on a circular array and does not need to be locked.

[0107] A data set acquisition module 303, based on the lock-free queue, acquires a first data set that meets a preset computing requirement and a second data set that meets a preset loading requirement from the classified data sets;

[0108] The data processing module 304 is used to distribute the target data in the first data set to a preset message publishing system based on the lock-free queue, and load the target data in the second data set to a preset distributed file system.

[0109] In the embodiment of the present invention, the data is put into module 302, which is used to:

[0110] Based on preset cleaning rules, data cleaning is performed on the target data in the lock-free queue, and based on preset classification rules and data features of the cleaned target data, the cleaned target data is classified to obtain multiple classified data sets.

[0111] In an embodiment of the present invention, the device further includes:

[0112] An information acquisition module, used to acquire data information of the target data;

[0113] A rule determination module is used to determine the preset cleaning rule based on the data information.

[0114] In the embodiment of the present invention, the data processing module 304 is used to:

[0115] Based on the asynchronous loading method, the target data in the second data set is loaded into the preset distributed file system.

[0116] In the embodiment of the present invention, the data processing module 304 is used to:

[0117] Based on a preset loading cycle and a preset loading volume, the target data in the second data set is loaded into the preset distributed file system.

[0118] In the embodiment of the present invention, the data acquisition module 301 is used to:

[0119] The queue capacity of the lock-free queue is obtained, and based on the queue capacity, target data for the target service is collected through the preset data interface.

[0120] The embodiment of the present invention provides a data processing device, which collects target data for a target service based on a preset data interface, wherein the target data includes signaling data and / or user plane data for the target service, puts the target data into a preset lock-free queue, and classifies the target data based on preset classification rules and data characteristics of the target data to obtain multiple classified data sets, obtains a first data set that meets the preset computing requirements and a second data set that meets the preset loading requirements from the classified data sets based on the lock-free queue, and the lock-free queue is a queue built based on a ring array and does not need to be locked, and distributes the target data in the first data set to a preset message publishing system based on the lock-free queue, and loads the target data in the second data set to a preset distributed file system. In this way, through the architecture design of the lock-free queue, the lock-free and non-blocking characteristics of the ring array can be fully utilized, and the target data in the first data set that meets the preset computing requirements in the collected target data can be distributed to the preset message publishing system, and the target data in the second data set that meets the preset loading requirements can be loaded into the preset distributed file system, which can improve the collection, processing, distribution and loading efficiency of the target data, reduce data latency, reduce resource consumption, and improve the data processing efficiency when distributing and loading data.

[0121] Embodiment 4

[0122] Figure 4 A schematic diagram of the hardware structure of an electronic device for implementing various embodiments of the present invention is provided.

[0123] The electronic device 400 includes but is not limited to: a radio frequency unit 401, a network module 402, an audio output unit 403, an input unit 404, a sensor 405, a display unit 404, a user input unit 407, an interface unit 408, a memory 409, a processor 410, and a power supply 411. Those skilled in the art will appreciate that Figure 4The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and the electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. In the embodiments of the present invention, the electronic device includes but is not limited to a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted terminal, a wearable electronic device, and a pedometer.

[0124] Among them, the processor 410 is used to: based on a preset data interface, collect target data for the target business, the target data includes signaling data and / or user plane data for the target business; put the target data into a preset lock-free queue, and classify the target data based on preset classification rules and data characteristics of the target data to obtain multiple classified data sets, the lock-free queue is a queue built based on a circular array and does not need to be locked; based on the lock-free queue, obtain a first data set in the classified data set that meets the preset computing requirements and a second data set that meets the preset loading requirements; based on the lock-free queue, distribute the target data in the first data set to a preset message publishing system, and load the target data in the second data set to a preset distributed file system.

[0125] In addition, the processor 410 is also used to: perform data cleaning processing on the target data in the lock-free queue based on preset cleaning rules, and classify the cleaned target data based on preset classification rules and data features of the cleaned target data to obtain multiple classified data sets.

[0126] In addition, the processor 410 is further configured to: obtain data information of the target data; and determine the preset cleaning rule based on the data information.

[0127] In addition, the processor 410 is further configured to: load the target data in the second data set into the preset distributed file system based on an asynchronous loading method.

[0128] In addition, the processor 410 is further configured to: load the target data in the second data set into the preset distributed file system based on a preset loading cycle and a preset loading amount.

[0129] In addition, the processor 410 is further configured to: obtain a queue capacity of the lock-free queue, and based on the queue capacity, collect target data for the target service through the preset data interface.

[0130] In addition, the processor 410 is also used to: obtain the queue capacity of the lock-free queue and the data information of the target data; determine the data insertion speed based on the queue capacity of the lock-free queue and the data information of the target data, and put the target data into a preset lock-free queue based on the data insertion speed.

[0131] An embodiment of the present invention provides an electronic device, which collects target data for a target service based on a preset data interface, wherein the target data includes signaling data and / or user plane data for the target service, puts the target data into a preset lock-free queue, and classifies the target data based on preset classification rules and data features of the target data to obtain multiple classified data sets, obtains a first data set that meets a preset computing requirement and a second data set that meets a preset loading requirement from the classified data set based on the lock-free queue, and the lock-free queue is a queue constructed based on a ring array and does not need to be locked, and distributes the target data in the first data set to a preset message publishing system based on the lock-free queue, and loads the target data in the second data set to a preset distributed file system. In this way, through the architecture design of the lock-free queue, the lock-free and non-blocking characteristics of the ring array can be fully utilized, and the target data in the first data set that meets the preset computing requirement in the collected target data can be distributed to the preset message publishing system, and the target data in the second data set that meets the preset loading requirement can be loaded into the preset distributed file system, which can improve the collection, processing, distribution and loading efficiency of the target data, reduce data latency, reduce resource consumption, and improve the data processing efficiency when distributing and loading data.

[0132] It should be understood that in the embodiment of the present invention, the radio frequency unit 401 can be used for receiving and sending signals during information transmission or calls. Specifically, after receiving downlink data from the base station, it is sent to the processor 410 for processing; in addition, the uplink data is sent to the base station. Generally, the radio frequency unit 401 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier, a duplexer, etc. In addition, the radio frequency unit 401 can also communicate with the network and other electronic devices through a wireless communication system.

[0133] The electronic device provides users with wireless broadband Internet access through the network module 402, such as helping users to send and receive emails, browse web pages, and access streaming media.

[0134] The audio output unit 403 can convert the audio data received by the RF unit 401 or the network module 402 or stored in the memory 409 into an audio signal and output it as sound. Moreover, the audio output unit 403 can also provide audio output related to a specific function performed by the electronic device 400 (for example, a call signal reception sound, a message reception sound, etc.). The audio output unit 403 includes a speaker, a buzzer, a receiver, etc.

[0135] The input unit 404 is used to receive audio or video signals. The input unit 404 may include a graphics processor (GPU) 4041 and a microphone 4042, and the graphics processor 4041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The processed image frame can be displayed on the display unit 404. The image frame processed by the graphics processor 4041 can be stored in the memory 409 (or other storage medium) or sent via the radio frequency unit 401 or the network module 402. The microphone 4042 can receive sound and can process such sound into audio data. The processed audio data can be converted into a format output that can be sent to a mobile communication base station via the radio frequency unit 401 in the case of a telephone call mode.

[0136] The electronic device 400 also includes at least one sensor 405, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor includes an ambient light sensor and a proximity sensor, wherein the ambient light sensor can adjust the brightness of the display panel 4061 according to the brightness of the ambient light, and the proximity sensor can turn off the display panel 4061 and / or the backlight when the electronic device 400 is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in each direction (generally three axes), and can detect the magnitude and direction of gravity when stationary, which can be used to identify the posture of the electronic device (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; the sensor 405 can also include a fingerprint sensor, a pressure sensor, an iris sensor, a molecular sensor, a gyroscope, a barometer, a hygrometer, a thermometer, an infrared sensor, etc., which will not be repeated here.

[0137] The display unit 406 is used to display information input by the user or information provided to the user. The display unit 406 may include a display panel 4061, which may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.

[0138] The user input unit 407 can be used to receive input digital or character information, and to generate key signal input related to user settings and function control of the electronic device. Specifically, the user input unit 407 includes a touch panel 4071 and other input electronic devices 4072. The touch panel 4071, also known as a touch screen, can collect the user's touch operation on or near it (such as the user's operation on the touch panel 4071 or near the touch panel 4071 using any suitable object or accessory such as a finger, stylus, etc.). The touch panel 4071 may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into the contact point coordinates, and then sends it to the processor 410, receives the command sent by the processor 410 and executes it. In addition, the touch panel 4071 can be implemented using multiple types such as resistive, capacitive, infrared and surface acoustic waves. In addition to the touch panel 4071, the user input unit 407 may also include other input electronic devices 4072. Specifically, other input electronic devices 4072 may include but are not limited to a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be elaborated here.

[0139] Furthermore, the touch panel 4071 may be overlaid on the display panel 4061. When the touch panel 4071 detects a touch operation on or near it, it is transmitted to the processor 410 to determine the type of the touch event. Then, the processor 410 provides a corresponding visual output on the display panel 4061 according to the type of the touch event. Figure 4 In the figure, the touch panel 4071 and the display panel 4061 are two independent components to realize the input and output functions of the electronic device. However, in some embodiments, the touch panel 4071 and the display panel 4061 can be integrated to realize the input and output functions of the electronic device, which is not limited here.

[0140] The interface unit 408 is an interface for connecting an external device to the electronic device 400. For example, the external device may include a wired or wireless headset port, an external power supply (or battery charger) port, a wired or wireless data port, a memory card port, a port for connecting a device with an identification module, an audio input / output (I / O) port, a video I / O port, a headphone port, etc. The interface unit 408 may be used to receive input (e.g., data information, power, etc.) from an external device and transmit the received input to one or more elements within the electronic device 400 or may be used to transmit data between the electronic device 400 and an external device.

[0141] The memory 409 can be used to store software programs and various data. The memory 409 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory 409 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0142] The processor 410 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. It executes various functions of the electronic device and processes data by running or executing software programs and / or modules stored in the memory 409, and calling data stored in the memory 409, so as to monitor the electronic device as a whole. The processor 410 may include one or more processing units; preferably, the processor 410 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, etc., and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 410.

[0143] The electronic device 400 may also include a power supply 411 (such as a battery) for supplying power to each component. Preferably, the power supply 411 may be logically connected to the processor 410 through a power management system, thereby implementing functions such as charging, discharging, and power consumption management through the power management system.

[0144] Preferably, an embodiment of the present invention further provides an electronic device, comprising a processor 410, a memory 409, and a computer program stored in the memory 409 and executable on the processor 410. When the computer program is executed by the processor 410, each process of the above-mentioned data processing method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0145] Embodiment 5

[0146] The embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, each process of the above-mentioned data processing method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0147] The embodiment of the present invention provides a computer-readable storage medium, which collects target data for a target service based on a preset data interface, wherein the target data includes signaling data and / or user plane data for the target service, puts the target data into a preset lock-free queue, and classifies the target data based on preset classification rules and data features of the target data to obtain multiple classified data sets, obtains a first data set that meets the preset computing requirements and a second data set that meets the preset loading requirements from the classified data sets based on the lock-free queue, and the lock-free queue is a queue constructed based on a ring array and does not need to be locked, and distributes the target data in the first data set to a preset message publishing system based on the lock-free queue, and loads the target data in the second data set to a preset distributed file system. In this way, through the architecture design of the lock-free queue, the lock-free and non-blocking characteristics of the ring array can be fully utilized, and the target data in the first data set that meets the preset computing requirements in the collected target data can be distributed to the preset message publishing system, and the target data in the second data set that meets the preset loading requirements can be loaded into the preset distributed file system, which can improve the collection, processing, distribution and loading efficiency of the target data, reduce data latency, reduce resource consumption, and improve the data processing efficiency when distributing and loading data.

[0148] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0149] The present invention is described with reference to flowcharts and / or block diagrams of methods, electronic devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing electronic device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing electronic device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0150] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing electronic device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0151] These computer program instructions can also be loaded onto a computer or other programmable data processing electronic device so that a series of operating steps are executed on the computer or other programmable electronic device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable electronic device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0152] In a typical configuration, a computing electronic device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0153] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0154] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage electronic devices or any other non-transmission media that can be used to store information that can be accessed by computing electronic devices. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0155] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or electronic device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or electronic device. In the absence of further restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or electronic device including the elements.

[0156] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0157] The above description is only an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A data processing method, characterized in that: The method comprises: Based on a preset data interface, collecting target data for a target service, the target data including signaling data and / or user plane data for the target service; The target data is placed in a preset lock-free queue, and the target data is classified based on preset classification rules and data features of the target data to obtain multiple classified data sets, wherein the lock-free queue is a queue constructed based on a circular array and does not need to be locked; Based on the lock-free queue, obtaining a first data set that meets a preset computing requirement and a second data set that meets a preset loading requirement from the classified data sets; Based on the lock-free queue, the target data in the first data set is distributed to a preset message publishing system, and the target data in the second data set is loaded into a preset distributed file system.

2. The method according to claim 1, characterized in that The target data is classified based on the preset classification rules and the data features of the target data to obtain multiple classification data sets, including: Based on preset cleaning rules, data cleaning is performed on the target data in the lock-free queue, and based on preset classification rules and data features of the cleaned target data, the cleaned target data is classified to obtain multiple classified data sets.

3. The method according to claim 2, characterized in that Before performing data cleaning processing on the target data in the lock-free queue based on the preset cleaning rule, the method further includes: Acquiring data information of the target data; Based on the data information, the preset cleaning rule is determined.

4. The method according to claim 1, characterized in that: The step of loading the target data in the second data set into a preset distributed file system includes: Based on the asynchronous loading method, the target data in the second data set is loaded into the preset distributed file system.

5. The method according to claim 4, characterized in that The step of loading the target data in the second data set into the preset distributed file system includes: Based on a preset loading cycle and a preset loading volume, the target data in the second data set is loaded into the preset distributed file system.

6. The method according to claim 1, characterized in that The collecting target data for the target business based on the preset data interface includes: The queue capacity of the lock-free queue is obtained, and based on the queue capacity, target data for the target service is collected through the preset data interface.

7. The method according to claim 1, characterized in that The step of placing the target data into a preset lock-free queue includes: Obtaining the queue capacity of the lock-free queue and data information of the target data; Based on the queue capacity of the lock-free queue and the data information of the target data, a data putting speed is determined, and based on the data putting speed, the target data is put into a preset lock-free queue.

8. A data processing device, characterized in that: The device comprises: A data collection module, used to collect target data for a target service based on a preset data interface, wherein the target data includes signaling data and / or user plane data for the target service; A data placing module is used to place the target data into a preset lock-free queue, and classify the target data based on preset classification rules and data features of the target data to obtain multiple classified data sets, wherein the lock-free queue is a queue constructed based on a circular array and does not need to be locked; A data set acquisition module, based on the lock-free queue, acquires a first data set that meets a preset computing requirement and a second data set that meets a preset loading requirement from the classified data sets; A data processing module is used to distribute the target data in the first data set to a preset message publishing system based on the lock-free queue, and to load the target data in the second data set to a preset distributed file system.

9. An electronic device, characterized in that: The method comprises a processor, a memory and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps of the data processing method according to any one of claims 1 to 7 when executed by the processor.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the data processing method according to any one of claims 1 to 7 are implemented.

Images