A data processing method, data server, medium and system

Abstract

This application provides a data processing method, data server, medium, and system. The method includes receiving real-time streaming data from a messaging system, the real-time streaming data including: a user identifier and real-time streaming information corresponding to the user identifier; retrieving offline tag data and / or real-time tag data corresponding to the user identifier from locally stored tag data; performing a hash operation on the user identifier to obtain a hash value corresponding to the user identifier, so as to retrieve a computing server corresponding to the hash value from a computing server cluster; allocating the real-time streaming data corresponding to the user identifier, as well as the offline tag data and / or real-time tag data, to the computing server for association processing, and deleting the offline tag data and / or real-time tag data corresponding to the user identifier locally. This solves the problem of low association efficiency between real-time streaming data and tag data caused by the backlog of large amounts of streaming data in the prior art.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data processing method, data server, medium and system. Background Technology

[0002] To implement services such as electronic fencing and user network service preference analysis, it is necessary to associate real-time user streaming data with tag data containing user attribute information for user screening in order to perform related services. In existing technologies, a computing server cluster obtains real-time streaming data from a messaging system and queries a tag database cluster for the tag data of the user corresponding to the real-time streaming data to establish the association between the real-time streaming data and the tag data.

[0003] However, real-time streaming data is transmitted from the messaging system to the computing server cluster at a rate of millions of messages per second. Every time the computing server cluster receives a real-time streaming data message, it needs to send a query request to the tag database cluster. Such a huge query demand often requires the tag database cluster to have hardware support of hundreds of databases. Even so, the massive amount of real-time streaming data still cannot be correlated and processed in a timely manner, resulting in a large backlog of streaming data and extremely low correlation efficiency between real-time streaming data and tag data. Summary of the Invention

[0004] This application provides a data processing method, data server, medium, and system to solve the problem of low correlation efficiency between real-time streaming data and tag data caused by the backlog of a large amount of streaming data in the prior art.

[0005] In a first aspect, this application provides a data processing method, comprising: receiving real-time streaming data from a messaging system, the real-time streaming data including: a user identifier and real-time streaming information corresponding to the user identifier; obtaining offline tag data and / or real-time tag data corresponding to the user identifier from locally stored tag data; performing a hash operation on the user identifier to obtain a hash value corresponding to the user identifier, so as to obtain a computing server corresponding to the hash value from a computing server cluster; allocating the real-time streaming data corresponding to the user identifier, as well as the offline tag data and / or real-time tag data, to the computing server for association processing, and deleting the offline tag data and / or real-time tag data corresponding to the user identifier locally.

[0006] In one specific implementation, the step of performing a hash operation on the user identifier to obtain the hash value corresponding to the user identifier includes: performing binary processing on the user identifier to obtain the processed user identifier; and performing a hash operation on the processed user identifier using a grouping or partitioning method to obtain the corresponding hash value.

[0007] In one specific implementation, the step of performing a hash operation on the processed user identifier using a grouping or partitioning method to obtain the corresponding hash value includes: obtaining the corresponding hash value n1 based on the processed user identifier m using the formula: n1 = murmurHash(hash(m)) % 256; or obtaining the corresponding hash value n2 based on the processed user identifier m using the formula: n2 = (murmurHash(hash(m)) % 256) * k / 256; where k is the number of computing servers in the computing server cluster, and k is a positive integer.

[0008] In one specific implementation, the data processing method further includes: obtaining real-time tag data from the messaging system and storing it locally.

[0009] In one specific embodiment, the data processing method further includes: retrieving offline tag data from the tag database at preset intervals and storing it locally.

[0010] Secondly, this application provides a data server, comprising: an acquisition module, configured to receive real-time streaming data from a messaging system, the real-time streaming data including: a user identifier and real-time streaming information corresponding to the user identifier; the acquisition module is further configured to acquire offline tag data and / or real-time tag data corresponding to the user identifier from locally stored tag data; a processing module, configured to perform a hash operation on the user identifier to obtain a hash value corresponding to the user identifier, so as to acquire a computing server corresponding to the hash value from a computing server cluster; the processing module is further configured to allocate the real-time streaming data corresponding to the user identifier, as well as the offline tag data and / or real-time tag data, to the computing server for association processing, and to delete the offline tag data and / or real-time tag data corresponding to the user identifier locally.

[0011] In one specific implementation, the processing module is specifically used to: perform binary processing on the user identifier to obtain the processed user identifier; and perform a hash operation on the processed user identifier using a grouping or partitioning method to obtain the corresponding hash value.

[0012] In one specific implementation, the processing module is specifically used to: obtain the corresponding hash value n1 based on the processed user identifier m using the formula: n1 = murmurHash(hash(m)) % 256; or, obtain the corresponding hash value n2 based on the processed user identifier m using the formula: n2 = (murmurHash(hash(m)) % 256) * k / 256; where k is the number of computing servers in the computing server cluster, and k is a positive integer.

[0013] In one specific implementation, the acquisition module is further configured to: acquire real-time tag data from the messaging system and store it locally.

[0014] In one specific implementation, the acquisition module is further configured to: acquire offline tag data from the tag database at preset intervals and store it locally.

[0015] Thirdly, this application provides a data server, including: a processor, a memory, and a communication interface; the memory is used to store executable instructions of the processor; wherein the processor is configured to execute the data processing method described in the first aspect by executing the executable instructions.

[0016] Fourthly, this application provides a readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data processing method described in the first aspect.

[0017] Fifthly, this application provides a data system, including: a messaging system, a computing server cluster, a tag database, and a data server as described in the second to fourth aspects.

[0018] This application provides a data processing method, a data server, a medium, and a system. The data processing method includes: receiving real-time streaming data from a messaging system, the real-time streaming data including a user identifier and real-time streaming information corresponding to the user identifier; obtaining offline tag data and / or real-time tag data corresponding to the user identifier from locally stored tag data; performing a hash operation on the user identifier to obtain a hash value corresponding to the user identifier, so as to obtain a computing server corresponding to the hash value from a computing server cluster; allocating the real-time streaming data corresponding to the user identifier, as well as the offline tag data and / or real-time tag data, to the computing server for association processing, and deleting the offline tag data and / or real-time tag data corresponding to the user identifier locally. Compared to existing technologies where the computing server queries tag data from a tag database cluster to correlate real-time streaming data upon receiving it, this application's data server stores tag data locally. Upon receiving real-time streaming data, it performs a hash operation on the user identifier to obtain the corresponding hash value. The tag data and real-time streaming data are then distributed to the computing servers within the computing server cluster corresponding to the hash values ​​for correlation processing. This effectively alleviates the immense query pressure on the tag database caused by massive amounts of real-time streaming data, avoids data backlog, and improves the correlation efficiency between real-time streaming data and tag data. It solves the problem of low correlation efficiency between real-time streaming data and tag data caused by large backlogs of streaming data in existing technologies. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a diagram of the data system architecture in the existing technology;

[0021] Figure 2 A data system architecture diagram provided in this application;

[0022] Figure 3 A flowchart illustrating an embodiment of a data processing method provided in this application;

[0023] Figure 4 A flowchart illustrating a second embodiment of a data processing method provided in this application;

[0024] Figure 5 This application provides a schematic diagram of the structure of a data server embodiment.

[0025] Figure 6 This is a schematic diagram of another data server embodiment provided in this application. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments made by those skilled in the art under the guidance of these embodiments are within the scope of protection of this application.

[0027] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0028] First, let me explain the terms used in this application:

[0029] Real-time streaming data refers to real-time log messages, including user identifiers and the corresponding real-time streaming information. Real-time streaming information can be signaling data periodically reported by user terminals, such as location updates, SMS messages, voice calls, or network access information.

[0030] Tag data includes user identifiers and corresponding attribute information. Attribute information may include the user's age, gender, location, and account balance. There are two types of tag data: offline tag data and real-time tag data. Offline tag data contains attribute information representing the user's fixed attributes, such as age, gender, and location; real-time tag data contains attribute information representing the user's changing attributes, such as account balance.

[0031] Figure 1 This is a diagram of the data system architecture in existing technologies. For example... Figure 1 As shown, the data system includes a message system 11, a tag database cluster 12, and a computing server cluster 13. The message system 11 can be a big data messaging middleware like Kafka, which obtains real-time streaming data and real-time tag data from various user terminals at base stations. The tag database cluster 12 includes multiple tag databases, which can be a distributed open-source database (Hadoop database, HBase) or a remote dictionary server (Redis). The tag database cluster 12 obtains real-time tag data from the message system 11 and merges it with its stored offline tag data to provide tag data query services for the computing server cluster 13. The computing server cluster 13 includes multiple computing servers. Each computing server obtains real-time streaming data from the message system 11 and, based on the user identifier corresponding to the real-time streaming data, initiates a tag data query request to the tag database cluster 12. The tag database cluster 12 responds to the query request, returning the tag data corresponding to the user identifier, enabling the computing server 13 to associate the real-time streaming data with the tag data, providing business data for subsequent applications such as electronic fencing and user network service preference analysis.

[0032] However, real-time streaming data is transmitted from the message system to the computing server cluster 13 at a rate of millions of messages per second. Every time the computing server cluster 13 receives a real-time streaming data message, it needs to send a query request to the tag database cluster 12. Such a huge query demand often requires the tag database cluster 12 to have hardware support of hundreds of databases. Even so, the massive amount of real-time streaming data still cannot be correlated and processed in a timely manner, resulting in a large backlog of streaming data and extremely low correlation efficiency between real-time streaming data and tag data.

[0033] Based on the above technical problems, the technical conception process of this application is as follows: How to alleviate the huge query pressure on the tag database caused by massive real-time streaming data on the computing server cluster, avoid the backlog of real-time streaming data, and improve the correlation efficiency between real-time streaming data and tag data.

[0034] The data processing scheme of this application will be described in detail below.

[0035] Figure 2 A data system architecture diagram is provided for this application, such as Figure 2 As shown, the data system may include: a messaging system 11, a data server 21, a tag database 22, and a computing server cluster 13. The messaging system 11 may be the big data messaging middleware Kafka, which obtains real-time streaming data and real-time tag data from various user terminals at the base station. The tag database 22 may be the distributed open-source database HBase or the data warehouse tool Hive, storing offline tag data.

[0036] Data server 21 retrieves offline tag data from tag database 22 and real-time tag data from message system 11, and stores them locally. Data server 21 retrieves real-time streaming data from message system 11, performs a hash operation on the user identifier in the real-time streaming data to obtain the hash value corresponding to the user identifier, and then retrieves the computing server corresponding to the hash value from the computing server cluster. The real-time streaming data corresponding to the user identifier, as well as the offline tag data and / or real-time tag data, are then allocated to the computing server.

[0037] The computing server cluster 13 includes multiple computing servers. The computing servers obtain real-time streaming data and tag data from the data server 21, and associate the real-time streaming data with the tag data to provide business data for subsequent business such as electronic fences and user network service preference analysis.

[0038] The technical solution of this application will now be described in detail through specific embodiments. It should be noted that the following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0039] Figure 3 A schematic flowchart illustrating an embodiment of a data processing method provided in this application. See also... Figure 3 The data processing method specifically includes the following steps:

[0040] Step S301: Receive real-time stream data from the messaging system. The real-time stream data includes: a user identifier and the real-time stream information corresponding to the user identifier.

[0041] In this embodiment, the real-time streaming data includes a user identifier and the corresponding real-time streaming information. The user identifier can be a user's mobile phone number. The real-time streaming information can be signaling data periodically reported by the user terminal, such as location update information, SMS messages, voice call information, or network access information.

[0042] Step S302: Obtain the offline tag data and / or real-time tag data corresponding to the user identifier from the locally stored tag data.

[0043] In this embodiment, the data server locally stores tag data. Tag data includes a user identifier and corresponding attribute information. Attribute information may include the user's age, gender, location, and account balance, among other things.

[0044] There are two types of tag data: offline tag data and real-time tag data. The attribute information in offline tag data represents the user's fixed attribute information, such as age, gender, and location; the attribute information in real-time tag data represents the user's changing attribute information, such as account balance.

[0045] Step S303: Perform a hash operation on the user identifier to obtain the hash value corresponding to the user identifier, so as to obtain the computing server corresponding to the hash value from the computing server cluster.

[0046] Step S304: Assign the real-time streaming data corresponding to the user identifier, as well as the offline tag data and / or real-time tag data, to the computing server for association processing, and delete the offline tag data and / or real-time tag data corresponding to the user identifier locally.

[0047] In this embodiment, the computing server cluster includes multiple computing servers that perform correlation processing on real-time streaming data and tag data. The data server performs hash calculation on the user identifier to obtain the hash value corresponding to the user identifier, and then retrieves the computing server corresponding to the hash value from the computing server cluster.

[0048] The data server assigns the real-time streaming data and tag data corresponding to the user identifier to the computing server corresponding to the hash value for association processing.

[0049] After real-time streaming data and tag data are distributed to the corresponding computing servers, the data servers delete the tag data corresponding to the user identifier locally, effectively saving local storage space and providing a prerequisite for efficient processing of massive real-time streaming data.

[0050] In this embodiment, the data server receives real-time streaming data from the messaging system. The real-time streaming data includes a user identifier and the corresponding real-time streaming information. It retrieves offline tag data and / or real-time tag data corresponding to the user identifier from locally stored tag data. It performs a hash operation on the user identifier to obtain a hash value, and then retrieves the computing server corresponding to the hash value from the computing server cluster. The real-time streaming data corresponding to the user identifier, along with the offline tag data and / or real-time tag data, are allocated to the computing server for association processing. Finally, the offline tag data and / or real-time tag data corresponding to the user identifier are deleted locally. Compared to existing technologies where the computing server queries tag data from a tag database cluster to correlate real-time streaming data upon receiving it, this application's data server stores tag data locally. Upon receiving real-time streaming data, it performs a hash operation on the user identifier to obtain a hash value. This hash value is then used to distribute the tag data and real-time streaming data to the corresponding computing server within the computing server cluster for correlation processing. This effectively alleviates the immense query pressure on the tag database caused by massive amounts of real-time streaming data, avoids data backlog, and improves the correlation efficiency between real-time streaming data and tag data. It solves the problem of low correlation efficiency between real-time streaming data and tag data caused by large backlogs of streaming data in existing technologies.

[0051] Figure 4 A flowchart illustrating a second embodiment of a data processing method provided in this application, in the above... Figure 3 Based on the illustrated embodiment, see also Figure 4 The data processing method specifically includes the following steps:

[0052] Step S401: Receive real-time stream data from the messaging system. The real-time stream data includes: a user identifier and the real-time stream information corresponding to the user identifier.

[0053] In this embodiment, the data server obtains real-time streaming data from the messaging system, performs data transcoding on the streaming data, and converts the binary data into text data. For example, the data structure of the text-formatted real-time streaming data can be: mobile phone number|balance. For instance, the real-time streaming data could be "18888888888|30".

[0054] In this embodiment, a window period can be set to merge multiple real-time stream data corresponding to the same user identifier acquired within the window period. The merged real-time stream data is then distributed to the corresponding computing servers for association processing, thereby further improving the association efficiency. For example, the window period can be 20 seconds.

[0055] In this embodiment, real-time streaming data can also be verified based on user identifiers to clean up illegal data. For example, real-time streaming data corresponding to illegal phone numbers can be deleted based on the user's phone number, and the cleaned real-time streaming data can be distributed to corresponding computing servers for association processing to further improve association efficiency.

[0056] Step S402: Obtain the offline tag data and / or real-time tag data corresponding to the user identifier from the locally stored tag data.

[0057] In this embodiment, before receiving real-time streaming data, the data server obtains real-time tag data from the messaging system and stores it locally; every preset period, it obtains offline tag data from the tag database and stores it locally.

[0058] Specifically, the data server can retrieve offline tag data from the tag database at preset intervals and store it locally. For example, the preset interval can be one day.

[0059] Specifically, the data server can use the distributed data acquisition and exchange tool FlinkX to export offline tag data files from the tag database. This tool can update the offline tag data files and synchronize data between the data server and the tag database. For example, the offline tag data files exported by the data server from the tag database are HFILE files or binary data files. The data server then converts the format of the offline tag data files to obtain text file format offline tag data files.

[0060] The data server extracts offline tag data from an offline tag data file in text file format. Specifically, the offline tag data includes a user identifier and corresponding fixed attribute information, such as age, gender, and location. For example, the data structure of the offline tag data can be: mobile phone number|age|gender|location. For instance, the offline tag data could be "18888888888|25|male|Beijing".

[0061] In this embodiment, the data server can obtain real-time tag data from the messaging system. Exemplarily, the real-time tag data obtained by the data server from the messaging system is binary data. The data server performs data transcoding on the real-time tag data, converting the binary data into text data. Specifically, the real-time tag data includes a user identifier and corresponding change attribute information, such as account balance. Exemplarily, the data structure of the real-time tag data can be: mobile phone number|account balance. For example, the real-time tag data can be "18888888888|30".

[0062] In this embodiment, the data server can merge offline data tags and real-time data tags based on the user identifier. For example, the user identifier can be a mobile phone number. The data server merges the offline tag data "18888888888|25|Male|Beijing" and the real-time tag data "18888888888|30" into the tag data "18888888888|25|Male|Beijing|30" based on the user identifier "18888888888|25|Male|Beijing|30".

[0063] Step S403: Perform binary processing on the user identifier to obtain the processed user identifier; use a grouping or partitioning method to perform a hash operation on the processed user identifier to obtain the corresponding hash value, so as to obtain the computing server corresponding to the hash value from the computing server cluster.

[0064] In this embodiment, the data server performs binary processing on the user identifier to obtain the processed user identifier. Specifically, the user identifier 'a' is processed into binary using the formula: m = (short)(hash(a) & 0x7fff) + a, where m is the processed user identifier.

[0065] For example, the user identifier is a mobile phone number, a = "18888888888". The above formula is used to perform binary processing on the mobile phone number "18888888888", and the processed user identifier m is [89,-15,49,56,56,56,56,56,56,56,56,56,56,56,56].

[0066] In this embodiment, a grouping or partitioning method can be used to perform a hash operation on the processed user identifier to obtain the corresponding hash value, and then obtain the computing server corresponding to the hash value from the computing server cluster.

[0067] Depending on the processing volume of tag data and streaming data, the computing server cluster can include 256 computing servers. When the computing server cluster includes 256 computing servers, a grouping approach can be used to perform hash operations on the processed user identifiers to obtain the corresponding hash values.

[0068] Specifically, based on the processed user identifier m, the corresponding hash value n1 can be obtained using the formula: n1 = murmurHash(hash(m)) % 256.

[0069] For example, the processed user identifier m is [89,-15,49,56,56,56,56,56,56,56,56,56,56]. Using the formula: n1 = murmurHash(hash(m)) % 256, the corresponding hash value n1 is 29.

[0070] A computing server cluster can include fewer than 256 computing servers. In this case, a partitioning approach can be used to perform a hash operation on the processed user identifier to obtain the corresponding hash value.

[0071] Specifically, based on the processed user identifier m, the corresponding hash value n2 can be obtained using the formula: n2 = (murmurHash(hash(m)) % 256) * k / 256; where k is the number of computing servers in the computing server cluster, and k is a positive integer.

[0072] Thus, when the number of computing servers in the computing server cluster changes, a partitioning approach is used to re-hash the processed user identifiers to obtain the corresponding hash values. This allows the data processing method in this embodiment to adapt well to the expansion or contraction of the computing server cluster.

[0073] Step S404: Assign the real-time streaming data corresponding to the user identifier, as well as the offline tag data and / or real-time tag data, to the computing server for association processing, and delete the offline tag data and / or real-time tag data corresponding to the user identifier locally.

[0074] In this embodiment, the same user identifier corresponds to the same hash value, and the same hash value corresponds to the same computing server. In other words, real-time streaming data and tag data corresponding to the same user identifier will be assigned to the same computing server for association processing.

[0075] In this embodiment, the data server performs binary processing on the user identifier, using a grouping or partitioning approach. It then performs a hash operation on the processed user identifier to obtain the corresponding hash value. This hash value is used to select the corresponding computing server from the computing server cluster. This ensures that real-time streaming data and tag data corresponding to the same user identifier are assigned to the same computing server for association processing, avoiding real-time streaming data backlog and further improving the association efficiency between real-time streaming data and tag data. Simultaneously, before receiving real-time streaming data, the data server retrieves real-time tag data from the messaging system and offline tag data from the tag database, storing them locally. This allows the data server to retrieve the offline tag data and / or real-time tag data corresponding to the user identifier from the locally stored tag data when receiving real-time streaming data. This further alleviates the enormous query pressure on the tag database caused by massive amounts of real-time streaming data on the computing server cluster, further improving the association efficiency between real-time streaming data and tag data.

[0076] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0077] Figure 5 This is a schematic diagram of the structure of a data server embodiment provided in this application; as shown below. Figure 5 As shown, the data server 50 includes an acquisition module 51 and a processing module 52. The acquisition module 51 receives real-time streaming data from the messaging system, including a user identifier and corresponding real-time streaming information. The acquisition module 51 also retrieves offline tag data and / or real-time tag data corresponding to the user identifier from locally stored tag data. The processing module 52 performs a hash operation on the user identifier to obtain a hash value, which is then used to retrieve the computing server corresponding to the hash value from the computing server cluster. The processing module 52 also allocates the real-time streaming data corresponding to the user identifier, as well as the offline tag data and / or real-time tag data, to the computing server for association processing, and deletes the offline tag data and / or real-time tag data corresponding to the user identifier locally.

[0078] The data server provided in this application embodiment can execute the technical solution shown in the above method embodiment. Its implementation principle and beneficial effects are similar, and will not be repeated here.

[0079] In one possible implementation, the processing module 52 is specifically used to perform binary processing on the user identifier to obtain the processed user identifier; and to perform a hash operation on the processed user identifier by grouping or partitioning to obtain the corresponding hash value.

[0080] In one possible implementation, the processing module 52 is specifically used to obtain the corresponding hash value n1 based on the processed user identifier m using the formula: n1 = murmurHash(hash(m)) % 256; or, based on the processed user identifier m, to obtain the corresponding hash value n2 using the formula: n2 = (murmurHash(hash(m)) % 256) * k / 256; where k is the number of computing servers in the computing server cluster, and k is a positive integer.

[0081] The data server provided in this application embodiment can execute the technical solution shown in the above method embodiment. Its implementation principle and beneficial effects are similar, and will not be repeated here.

[0082] In one possible implementation, the acquisition module 51 is further specifically configured to acquire real-time tag data from the messaging system and store it locally. The acquisition module 51 is also specifically configured to acquire offline tag data from the tag database at preset intervals and store it locally.

[0083] The data server provided in this application embodiment can execute the technical solution shown in the above method embodiment. Its implementation principle and beneficial effects are similar, and will not be repeated here.

[0084] Figure 6 A schematic diagram of another data server structure provided in this application. (See diagram below.) Figure 6 As shown, the server 60 includes a processor 61, a memory 62, and a communication interface 63; wherein the memory 62 is used to store executable instructions of the processor 61; the processor 61 is configured to execute the technical solutions in any of the foregoing method embodiments by executing the executable instructions.

[0085] Optionally, the memory 62 can be either standalone or integrated with the processor 61.

[0086] Optionally, when the memory 62 is a device independent of the processor 61, the server 60 may further include a bus 64 for connecting the aforementioned devices.

[0087] The server is used to execute the technical solutions in any of the aforementioned method embodiments. Its implementation principle and technical effect are similar, and will not be described again here.

[0088] This application also provides a readable storage medium storing a computer program thereon, which, when executed by a processor, implements the technical solutions provided in any of the foregoing embodiments.

[0089] This application also provides a data system, including a messaging system, a computing server cluster, a tag database, and a data server as provided in any of the foregoing embodiments.

[0090] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0091] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A data processing method, characterized in that, The method, applied to a data server, includes: Receive real-time stream data from the messaging system. The real-time stream data includes: a user identifier and real-time stream information corresponding to the user identifier. The real-time stream information is signaling data periodically reported by the user terminal. From the locally stored tag data, retrieve offline tag data and / or real-time tag data corresponding to the user identifier. The offline tag data in the locally stored tag data is obtained from the tag database, and the real-time tag data is obtained from the message system. The tag data includes the user identifier and the attribute information corresponding to the user identifier. The attribute information in the offline tag data represents the user's fixed attribute information. The attribute information in the real-time tag data represents the user's changing attribute information. The message system is the big data message middleware Kafka. Performing a hash operation on the user identifier to obtain a hash value corresponding to the user identifier, in order to obtain a computing server corresponding to the hash value from the computing server cluster, includes: performing binary processing on the user identifier to obtain a processed user identifier; performing a hash operation on the processed user identifier using a grouping or partitioning method to obtain a corresponding hash value; wherein, performing a hash operation on the processed user identifier using a grouping or partitioning method to obtain a corresponding hash value includes: grouping according to the processed user identifier m using the formula: Get the corresponding hash value When the computing server cluster consists of fewer than 256 computing servers, a partitioning method is used. The partitioning is based on the processed user identifier m, using the following formula: Get the corresponding hash value Where k is the number of computing servers in the computing server cluster, and k is a positive integer; The real-time streaming data corresponding to the user identifier, as well as the offline tag data and / or real-time tag data, are allocated to the computing server for association processing. The offline tag data and / or real-time tag data corresponding to the user identifier are deleted locally. The real-time streaming data and tag data corresponding to the same user identifier are allocated to the same computing server for association processing.

2. The data processing method according to claim 1, characterized in that, Also includes: Real-time tag data is obtained from the messaging system and stored locally.

3. The data processing method according to claim 2, characterized in that, Also includes: At preset intervals, offline tag data is retrieved from the tag database and stored locally.

4. A data server, characterized in that, include: The acquisition module is used to receive real-time stream data from the messaging system. The real-time stream data includes: a user identifier and real-time stream information corresponding to the user identifier. The real-time stream information is signaling data periodically reported by the user terminal. The acquisition module is further configured to acquire offline tag data and / or real-time tag data corresponding to the user identifier from locally stored tag data. The offline tag data in the locally stored tag data is acquired from a tag database, and the real-time tag data is acquired from the message system. The tag data includes a user identifier and attribute information corresponding to the user identifier. The attribute information in the offline tag data represents the user's fixed attribute information. The attribute information in the real-time tag data represents the user's changing attribute information. The message system is the big data message middleware Kafka. The processing module is configured to perform a hash operation on the user identifier to obtain a hash value corresponding to the user identifier, so as to obtain a computing server corresponding to the hash value from the computing server cluster. The module includes: performing binary processing on the user identifier to obtain a processed user identifier; and performing a hash operation on the processed user identifier using a grouping or partitioning method to obtain a corresponding hash value. The step of performing a hash operation on the processed user identifier using a grouping or partitioning method to obtain a corresponding hash value includes: grouping the processed user identifier m using the formula: Get the corresponding hash value When the computing server cluster consists of fewer than 256 computing servers, a partitioning method is used. The partitioning is based on the processed user identifier m, using the following formula: Get the corresponding hash value Where k is the number of computing servers in the computing server cluster, and k is a positive integer; The processing module is further configured to allocate the real-time stream data corresponding to the user identifier, as well as the offline tag data and / or real-time tag data, to the computing server for association processing, and to delete the offline tag data and / or real-time tag data corresponding to the user identifier locally, wherein the real-time stream data and tag data corresponding to the same user identifier are allocated to the same computing server for association processing.

5. The data server according to claim 4, characterized in that, The acquisition module is also specifically used for: Real-time tag data is obtained from the messaging system and stored locally.

6. The data server according to claim 5, characterized in that, The acquisition module is also specifically used for: At preset intervals, offline tag data is retrieved from the tag database and stored locally.

7. A data server, characterized in that, include: Processor, memory, communication interface; The memory is used to store the executable instructions of the processor; The processor is configured to perform a data processing method according to any one of claims 1 to 3 by executing the executable instructions.

8. A readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the data processing method according to any one of claims 1 to 3.

9. A data system, characterized in that, include: The messaging system, the computing server cluster, the tag database, and the data server as described in any one of claims 4-6.

Images