Data processing method and device, electronic equipment and computer readable storage medium

Abstract

Embodiments of the present application disclose a data processing method and device, electronic equipment and computer readable storage medium; in the embodiments of the present application, the original data is acquired, and the original data is divided to obtain a plurality of behavior data blocks;Adjacent candidate block behavior data is filtered from the plurality of behavior data blocks;The behavior data of the candidate adjacent user of the target user in the candidate block behavior data is acquired, the mapping relationship between the behavior data of the target user and the candidate adjacent user is generated, and the mapping relationship is stored in the cache;Adjacent subsequent block behavior data other than the candidate block behavior data is filtered from the plurality of behavior data blocks;The behavior data of the subsequent adjacent user of the subsequent user of the subsequent block behavior data is acquired, and the behavior data of the subsequent user and the subsequent adjacent user is replaced in the behavior data of the target user and the candidate adjacent user in the mapping relationship. The embodiments of the present application can save cache.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a data processing method and device, electronic equipment and computer readable storage medium. BACKGROUND

[0002] With the development of the Internet, more and more users use the Internet. In order to better serve the users, the behavior data of the users is generally statistically analyzed.

[0003] At present, when a user is analyzed, the behavior data of the adjacent users of the user is counted, and the counting result of the behavior data of the adjacent users is taken as the feature of the user. Therefore, when the behavior data of a user is obtained, the behavior data of the adjacent users of the user corresponding to the behavior data of the user is obtained, and then counted. This method needs to store the behavior data of the adjacent users of all users, resulting in a large amount of cache consumption. SUMMARY

[0004] The embodiments of the present application provide a data processing method and device, electronic equipment and computer readable storage medium, which can solve the technical problem of consuming a large amount of cache.

[0005] A data processing method comprises:

[0006] Obtaining original data, and dividing the original data to obtain a plurality of pieces of behavior data;

[0007] Filtering out adjacent candidate piece behavior data from the plurality of pieces of behavior data;

[0008] Obtaining the behavior data of a candidate adjacent user of a target user in the candidate piece behavior data, generating a mapping relationship between the behavior data of the target user and the candidate adjacent user, and storing the mapping relationship in a cache;

[0009] Filtering out adjacent subsequent piece behavior data other than the candidate piece behavior data from the plurality of pieces of behavior data;

[0010] Obtaining the behavior data of a subsequent adjacent user of a subsequent user of the subsequent piece behavior data, and replacing the behavior data of the target user and the candidate adjacent user in the mapping relationship with the behavior data of the subsequent user and the subsequent adjacent user.

[0011] Correspondingly, the embodiments of the present application provide a data processing device, comprising:

[0012] A first obtaining module is configured to obtain original data, and divide the original data to obtain a plurality of pieces of behavior data;

[0013] The first screening module is configured to screen out adjacent candidate block behavior data from the plurality of block behavior data.

[0014] The second obtaining module is configured to obtain behavior data of a candidate adjacent user of a target user in the candidate block behavior data, generate a mapping relationship between the behavior data of the target user and the candidate adjacent user, and store the mapping relationship in a cache.

[0015] The second screening module is configured to screen out adjacent subsequent block behavior data other than the candidate block behavior data from the plurality of block behavior data.

[0016] The third obtaining module is configured to obtain behavior data of a subsequent adjacent user of a subsequent user in the subsequent block behavior data, and replace the behavior data of the target user and the candidate adjacent user in the mapping relationship with the behavior data of the subsequent user and the subsequent adjacent user.

[0017] Optionally, the adjacent candidate block behavior data includes first block behavior data and second block behavior data, and the adjacent subsequent block behavior data includes third block behavior data and fourth block behavior data.

[0018] Correspondingly, the second obtaining module is specifically configured to perform:

[0019] obtain behavior data of a first candidate adjacent user of a first target user in the first block behavior data, and generate a first mapping relationship between the behavior data of the first target user and the first candidate adjacent user;

[0020] obtain behavior data of a second candidate adjacent user of a second target user in the second block behavior data, and generate a second mapping relationship between the behavior data of the second target user and the second candidate adjacent user;

[0021] store the first mapping relationship and the second mapping relationship in the cache.

[0022] The third obtaining module is specifically configured to perform:

[0023] obtain behavior data of a first subsequent adjacent user of a first subsequent user in the third block behavior data, and obtain behavior data of a second subsequent adjacent user of a second subsequent user in the fourth block behavior data;

[0024] replace the behavior data of the first target user and the first candidate adjacent user in the first mapping relationship with the behavior data of the first subsequent user and the first subsequent adjacent user;

[0025] Replace the behavior data of the second target user and the second candidate neighboring user in the second mapping relationship with the behavior data of the second subsequent user and the second subsequent neighboring user.

[0026] Optionally, the second obtaining module is specifically configured to perform:

[0027] Divide the candidate block behavior data into each candidate sub-block behavior data, and the difference between the number of each candidate sub-block behavior data is within a preset numerical interval.

[0028] Obtain the behavior data of the candidate neighboring user of the target user of each candidate sub-block behavior data from the external database each time until the behavior data of the candidate neighboring user of the target user of the candidate block behavior data is obtained.

[0029] Optionally, the second obtaining module is specifically configured to perform:

[0030] Divide the candidate block behavior data into each candidate sub-block behavior data by using a consistent hash algorithm.

[0031] Optionally, the mapping relationship is a state management function.

[0032] Correspondingly, the second obtaining module is specifically configured to perform:

[0033] Obtain the candidate neighboring user of the target user in the candidate block behavior data.

[0034] Store the identifier of the target user and the identifier of the candidate neighboring user in a first variable in the state management function.

[0035] Obtain the behavior data of the candidate neighboring user.

[0036] Store the identifier of the candidate neighboring user and the behavior data of the candidate neighboring user in a second variable in the state management function.

[0037] Store the state management function in a cache.

[0038] Optionally, the second obtaining module is specifically configured to perform:

[0039] Convert the data type of the identifier of the target user and the identifier of the candidate neighboring user into an integer type to obtain an integer type of the identifier of the target user and an integer type of the identifier of the candidate neighboring user.

[0040] Store the integer type of the identifier of the target user and the integer type of the identifier of the candidate neighboring user in the first variable in the state management function.

[0041] Optionally, the data processing apparatus further comprises:

[0042] a synchronization module configured to synchronize the candidate block behavior data to the executors in the execution servers.

[0043] a sending module configured to generate storage completion information and send the storage completion information to the driving server, so that the driving server determines the identifier of the executor in the execution server, and the executor corresponding to the identifier is configured to count the behavior data in the mapping relationship based on the candidate block behavior data.

[0044] In addition, the embodiment of the present application further provides an electronic device comprising a processor and a memory, wherein the memory stores a computer program, and the processor is configured to run the computer program in the memory to implement the data processing method provided by the embodiment of the present application.

[0045] In addition, the embodiment of the present application further provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is adapted to be loaded by a processor to execute any one of the data processing methods provided by the embodiment of the present application.

[0046] In addition, the embodiment of the present application further provides a computer program product comprising a computer program, wherein the computer program is executed by a processor to implement any one of the data processing methods provided by the embodiment of the present application.

[0047] In the embodiment of the present application, the original data is first acquired, and the original data is divided to obtain multiple pieces of behavior data. Then, adjacent candidate block behavior data is screened out from the multiple pieces of behavior data. Next, the behavior data of a candidate adjacent user of a target user in the candidate block behavior data is acquired, a mapping relationship between the behavior data of the target user and the candidate adjacent user is generated, and the mapping relationship is stored in a cache. Secondly, adjacent subsequent block behavior data other than the candidate block behavior data is screened out from the multiple pieces of behavior data. Finally, the behavior data of a subsequent adjacent user of a subsequent user of the subsequent block behavior data is acquired, and the behavior data of the subsequent user and the subsequent adjacent user is replaced with the behavior data of the target user and the candidate adjacent user in the mapping relationship.

[0048] That is, in the embodiment of the present application, the mapping relationship in the cache only stores the behavior data of the candidate adjacent user of the target user of the adjacent candidate block behavior data, or only stores the behavior data of the subsequent adjacent user of the subsequent user of the adjacent subsequent block behavior data, that is, the mapping relationship in the cache only stores the behavior data of the adjacent user of the user of the adjacent block data, without storing the behavior data of the adjacent user of all users in the original data, thereby saving the cache occupied by the behavior data of the adjacent user. BRIEF DESCRIPTION OF DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments will be briefly introduced. Obviously, the drawings in the following description only constitute some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort.

[0050] Figure 1 is a scene schematic diagram of a data processing process provided by an embodiment of the present application;

[0051] Figure 2 is a framework schematic diagram of Spark Streaming provided by an embodiment of the present application;

[0052] Figure 3 is a flow schematic diagram of a data processing method provided by an embodiment of the present application;

[0053] Figure 4 is a flow schematic diagram of another data processing method provided by an embodiment of the present application;

[0054] Figure 5 is a structure schematic diagram of a data processing apparatus provided by an embodiment of the present application;

[0055] Figure 6 is a structure schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0056] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments only constitute some embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative effort fall within the scope of protection of the present application.

[0057] The present application provides a data processing method, apparatus, electronic device and computer readable storage medium. The data processing apparatus can be integrated in an electronic device, which can be a server, a terminal or other devices.

[0058] The server can be a standalone physical server, a server cluster or a distributed system composed of multiple physical servers, a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, network acceleration services (Content Delivery Network, CDN), and basic cloud computing services such as big data and artificial intelligence platforms.

[0059] And, the plurality of servers can be composed as a blockchain, and the servers are nodes on the blockchain.

[0060] The terminal can be a smartphone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, a vehicle terminal, etc., but is not limited thereto. The terminal and the server can be connected directly or indirectly through wired or wireless communication, which is not limited in the present application.

[0061] For example, referring to Figure 1 For example, taking the case of integrating the data processing apparatus in the server, the server can be used to obtain raw data, and divide the raw data to obtain a plurality of pieces of behavior data; adjacent candidate piece of behavior data is filtered out from the plurality of pieces of behavior data; the behavior data of a candidate adjacent user of a target user in the candidate piece of behavior data is obtained, a mapping relationship between the behavior data of the target user and the candidate adjacent user is generated, and the mapping relationship is stored in a cache; adjacent subsequent piece of behavior data other than the candidate piece of behavior data is filtered out from the plurality of pieces of behavior data; the behavior data of a subsequent adjacent user of a subsequent user of the subsequent piece of behavior data is obtained, and the behavior data of the subsequent user and the subsequent adjacent user is replaced with the behavior data of the target user and the candidate adjacent user in the mapping relationship.

[0062] The embodiments of the present application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, intelligent transportation, and assisted driving.

[0063] In addition, "a plurality of" in the embodiments of the present application refers to two or more. "First" and "second" and the like in the embodiments of the present application are used to distinguish the description, and cannot be understood as implying relative importance.

[0064] It can be understood that in the specific embodiments of the present application, the behavior data of the user and other related data are involved, and when the above embodiments of the present application are applied to specific products or technologies, the user's permission or consent is required, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0065] Spark Streaming is a high-throughput real-time stream data processing technology with fault tolerance mechanism. The framework of Spark Streaming is as follows Figure 2As shown, the receiving server (the receiving server can also count the behavior data of the adjacent users) containing a receiver receives the original data and divides the original data to obtain multiple pieces of behavior data. The receiving server obtains the behavior data of the adjacent users of the user of each piece of behavior data and stores the behavior data of the adjacent users of all users of the original data in the cache. After storing the behavior data of the adjacent users of the user of each piece of behavior data, the receiving server synchronizes each piece of behavior data to the execution server, and the receiving server notifies the driving server. After receiving the notification, the driving server calls the executor in the execution server through the driver in the driving server to count the behavior data of the adjacent users corresponding to each piece of behavior data in the cache based on each piece of behavior data.

[0066] The following will be described in detail respectively. It should be noted that the description order of the following embodiments is not limited as the preferred order of the embodiments.

[0067] In this embodiment, the data processing device will be described from the perspective of the data processing device, which can be integrated in a server or a terminal or the like. In order to facilitate the description of the data processing method of the present application, the data processing device will be described in detail below by integrating the data processing device in the server, that is, the server is taken as the execution subject for detailed description.

[0068] Please refer to Figure 3 , Figure 3 is a flowchart of a data processing method provided by an embodiment of the present application. The data processing method can include:

[0069] S301, obtaining original data and dividing the original data to obtain multiple pieces of behavior data.

[0070] The original data can be the behavior data of the users under different systems collected by SDK. After the server obtains the original data, the original data is divided to obtain multiple pieces of behavior data. Optionally, the server can divide the original data according to a preset time interval to obtain multiple pieces of behavior data. For example, the server can divide the original data in the first second to the 200th second into one piece of behavior data.

[0071] Among them, the server can be a receiving server in Spark Streaming, then the server can obtain the original data through a receiver in the server, and divide the original data to obtain multiple pieces of behavior data.

[0072] It should be noted that the server can divide the original data while receiving the original data, or divide the original data after receiving the original data. In addition, after the backend server collects the original data through the SDK, the original data can be sent to Kafka, and Kafka classifies the original data through a preset topic. The server then obtains the original data in each topic of Kafka and parses the original data in each topic to obtain a data stream. Then the server divides the data stream to obtain multiple pieces of behavior data.

[0073] A data stream refers to a series of dynamic data collections that are infinite in time distribution and quantity. The value of the data in the collection decreases over time, so the data stream must be calculated in real time.

[0074] S302, filtering adjacent candidate block behavior data from the multiple pieces of behavior data.

[0075] After the server obtains the multiple pieces of behavior data, it filters adjacent candidate adjacent data from the multiple pieces of behavior data.

[0076] If the server divides the original data after receiving the original data, it can randomly filter adjacent candidate block behavior data from the multiple pieces of behavior data, or set a timestamp for each piece of behavior data according to the receiving time of each piece of behavior data, and then filter adjacent candidate block behavior data from the multiple pieces of behavior data based on the timestamp. The receiving time corresponding to the timestamp of the candidate block behavior data is earlier than the receiving time corresponding to the timestamp of the subsequent block behavior data in the multiple pieces of behavior data except the candidate block behavior data.

[0077] If the server divides the original data while receiving the original data, the first divided block behavior data can be directly used as the candidate block behavior data, without the need to filter adjacent candidate block behavior data from the multiple pieces of behavior data.

[0078] For example, the preset time interval is 200 seconds, the timing starts when the original data is received, and when the time reaches 200 seconds, the server groups the original data received in the first 200 seconds into the first block behavior data. Next, continue to receive the original data, and when the time reaches 400 seconds, group the original data received in the 201st to 400th seconds into the second block behavior data. Next, continue to receive the original data, and when the time reaches 600 seconds, group the original data received in the 401st to 600th seconds into the third block behavior data. At this time, the first block behavior data, the second block behavior data and the third block behavior data are directly used as the candidate block behavior data.

[0079] The number of adjacent candidate block behavior data can refer to two pieces of adjacent candidate block behavior data, for example, the adjacent candidate block behavior data can be the current candidate block behavior data and the previous candidate block behavior data of the current candidate block behavior data.

[0080] For example, the raw data received in the 201st to 400th second is grouped into the second block behavior data, and the current candidate block behavior data can be the second block behavior data. The raw data received in the 1st to 200th second is grouped into the first block behavior data, and the previous candidate block behavior data of the current candidate block behavior data can be the first block behavior data.

[0081] Alternatively, the number of adjacent candidate block behavior data can also refer to three pieces of adjacent candidate block behavior data, for example, the adjacent candidate block behavior data can be the current candidate block behavior data, the next candidate block behavior data of the current candidate block behavior data and the previous candidate block behavior data of the current candidate block behavior data.

[0082] For example, the raw data received in the 201st to 400th second is grouped into the second block behavior data, and the current candidate block behavior data can be the second block behavior data. The raw data received in the 1st to 200th second is grouped into the first block behavior data, and the previous candidate block behavior data of the current candidate block behavior data can be the first block behavior data. The raw data received in the 401st to 600th second is grouped into the third block behavior data, and the next candidate block behavior data of the current candidate block behavior data can be the third block behavior data.

[0083] The number of adjacent candidate block behavior data can be set by the user according to the actual situation, which is not limited in the present application.

[0084] S303, obtaining the behavior data of the candidate adjacent user of the target user in the candidate block behavior data, generating a mapping relationship between the behavior data of the target user and the candidate adjacent user, and storing the mapping relationship in the cache.

[0085] After the server obtains the candidate block behavior data, the target user corresponding to the candidate block behavior data is determined, and then the candidate adjacent user of the target user is queried from the external database, and the behavior data of the candidate adjacent user is queried from the external database.

[0086] When the server queries the behavior data of the candidate adjacent user from the external database in units of blocks, the query amount of the external database will increase sharply in a short time, which will cause the external database to be unstable.

[0087] To solve the technical problem, in some embodiments, obtaining the behavior data of the candidate adjacent user of the target user in the candidate block behavior data comprises:

[0088] The candidate block behavior data is divided into candidate sub-block behavior data, and the difference between the number of each candidate sub-block behavior data is within a preset numerical interval.

[0089] Each time, the behavior data of the candidate neighboring users of the target user of the candidate sub-block behavior data is obtained from the external database until the behavior data of the candidate neighboring users of the target user of the candidate block behavior data is obtained.

[0090] In this embodiment, the candidate block behavior data is divided into candidate sub-block behavior data, and the difference between the number of each candidate sub-block behavior data is within a preset numerical interval, and then the behavior data of the candidate neighboring users of the target user of the candidate sub-block behavior data is obtained from the external database each time. Since the difference between the number of each candidate sub-block behavior data is within a preset numerical interval, when the behavior data of the candidate neighboring users is queried from the external database each time, the query amount of the external database will not increase sharply in a short time, thereby ensuring the stability of the external database.

[0091] The method of dividing the candidate block behavior data into candidate sub-block behavior data can be selected according to actual conditions, for example, the candidate block behavior data can be divided into candidate sub-block behavior data by a hash algorithm or a consistent hash algorithm, which is not limited in this embodiment.

[0092] When the consistent hash algorithm is used to divide the candidate block behavior data into candidate sub-block behavior data, the process of dividing the candidate block behavior data into candidate sub-block behavior data can be that one candidate sub-block behavior data corresponds to one target address on a hash ring space, the original address of each behavior data in the candidate block behavior data on the hash ring space is calculated, and the behavior data is distributed to the candidate sub-block behavior data corresponding to the first target address encountered in the clockwise movement of the original address.

[0093] After the behavior data of the candidate neighboring users is obtained, the server then generates a mapping relationship between the target user data and the behavior data of the candidate neighboring users, and stores the mapping relationship in the cache.

[0094] The candidate neighboring users of the target user refer to the users corresponding to the network nodes within a preset range of the distance of the nodes corresponding to the target user in the relationship network of the target user.

[0095] The mapping relationship can be a state management function (MapWithState) or a hash table (HashMap). The cache can be a shared cache database.

[0096] When the mapping relationship is a state management function, behavior data of a candidate neighboring user of a target user in candidate block behavior data is obtained, a mapping relationship between the target user and the behavior data of the candidate neighboring user is generated, and the mapping relationship is stored in a cache, including:

[0097] A candidate neighboring user of a target user in candidate block behavior data is obtained.

[0098] An identifier of the target user and an identifier of the candidate neighboring user are stored in a first variable in the state management function.

[0099] Behavior data of the candidate neighboring user is obtained.

[0100] The identifier of the candidate neighboring user and the behavior data of the candidate neighboring user are stored in a second variable in the state management function.

[0101] The state management function is stored in the cache.

[0102] The server can obtain the candidate neighboring user of the target user in the candidate block behavior data from an external database, and the format of the obtained candidate neighboring user of the target user can be (id, friendids), where id represents the identifier of the target user, and friendids represents the identifier of the candidate neighboring user. The external database can be a Redis database.

[0103] The identifier of the candidate neighboring user and the behavior data of the candidate neighboring user can be stored in the second variable in the state management function in the form of (friendids, data).

[0104] Since the integer type (int) can save cache, and the identifier of the target user and the identifier of the candidate neighboring user do not exceed uint_32, in some possible implementation manners, the data types of the identifier of the target user and the identifier of the candidate neighboring user can be converted into the integer type, to obtain an integer type of the identifier of the target user and an integer type of the identifier of the candidate neighboring user, and then the integer type of the identifier of the target user and the integer type of the identifier of the candidate neighboring user are stored in the first variable in the state management function, thereby saving cache.

[0105] In Spark Streaming, the driver server is scheduled at the execution server level, specifically at the Node_Local level (where the behavioral data and the code that computes it reside on the same execution server but not in the same executor, i.e., not in the same process; one executor corresponds to one process). The driver server can only determine which execution server to call to analyze the behavioral data of candidate neighboring users; it doesn't determine which executor within that server will perform the analysis. Therefore, when multiple executors exist on an execution server, the behavioral data of candidate neighboring users for the same target user may not be analyzed by the same executor, leading to incorrect statistical results.

[0106] For example, candidate block behavior data includes a first block of behavior data and a second block of behavior data. Both the first and second blocks of behavior data include user 'a'. Execution server A includes executor 1 and executor 2. The driver server schedules at the execution server level, determining execution server A as the execution server for statistically analyzing the behavior data of the first candidate neighboring user of the first target user in the first block of behavior data. Execution server A then uses executor 1 to statistically analyze the behavior data of the first candidate neighboring user.

[0107] Since both the first and second pieces of behavioral data include user 'a', and the execution server for the first piece of behavioral data is execution server A, the driver server determines the behavioral data of the second candidate neighboring user of the second target user for the second piece of behavioral data to be execution server A (that is, at this time, the scheduling logic of the driver server is to assign the behavioral data of the neighboring users of the same user to the same server). However, execution server A may perform the statistical analysis on the behavioral data of the second candidate neighboring user through executor 2, which may lead to errors in the statistical results.

[0108] Therefore, in order to ensure that the behavior data of adjacent users of the same user are counted on the same executor, and thus to ensure that the statistical results are not erroneous, in some embodiments, after filtering out adjacent candidate blocks of behavior data from multiple blocks of behavior data, the method further includes:

[0109] The candidate block behavior data is synchronized to the executors of each execution server;

[0110] Accordingly, after storing the mapping relationship in the cache, the following is also included:

[0111] A storage completion message is generated and sent to the driver server so that the driver server can determine the identifier of the executor in the execution server. The executor corresponding to the identifier is used to perform statistics on the behavior data in the mapping relationship based on the candidate block behavior data.

[0112] In the embodiment, the server synchronizes the candidate block behavior data to the executors in each executor server, and after storing the mapping relationship in the cache, generates storage completion information and sends the storage completion information to the driver server, so that the driver server determines the identifier of the executor server and then determines the identifier of the executor.

[0113] The executor corresponding to the identifier of the executor finds the identifier of the candidate adjacent user corresponding to the identifier of the target user from the mapping relationship in the cache, then finds the behavior data corresponding to the identifier of the candidate adjacent user from the mapping relationship in the cache, and then performs statistics on the behavior data corresponding to the identifier of the candidate adjacent user, that is, performs cumulative summation on the behavior data corresponding to the identifier of the candidate adjacent user to obtain a statistical result, and takes the statistical result as a feature of the target user. Finally, the executor can periodically send the statistical result to the distributed file system (Hadoop Distributed File System, HDFS).

[0114] In the embodiment of the application, the driver server schedules at the level of the executor, that is, the driver server schedules at the Process_Local level (the Process_Local level means that the behavior data and the code for calculating the behavior data are in the same executor, that is, in the same process). The driver server can determine the executor that calculates the behavior data corresponding to the identifier of the candidate adjacent user, so that the driver server can assign the behavior data of the adjacent user of the same user to the same executor for calculation, thereby ensuring the accuracy of the statistical result.

[0115] S304, filtering out the adjacent subsequent block behavior data other than the candidate block behavior data from the multiple block behavior data.

[0116] The server filters out the adjacent subsequent block behavior data other than the candidate block behavior data from the multiple block behavior data after storing the mapping relationship in the cache.

[0117] The method of filtering out the adjacent subsequent block behavior data other than the candidate block behavior data from the multiple block behavior data can refer to the method of filtering out the adjacent candidate block behavior data from the multiple block behavior data, which will not be described here in the embodiment.

[0118] It should be noted that if the server receives the original data and divides the original data at the same time, the server directly takes the block behavior data obtained by dividing the original data received later as the subsequent block behavior data, and at this time, it is not necessary to filter out the subsequent block behavior data from the multiple block behavior data.

[0119] S305, obtaining the behavior data of the subsequent neighbor user of the subsequent user of the subsequent block behavior data, and replacing the behavior data of the target user and the candidate neighbor user in the mapping relationship with the behavior data of the subsequent user and the subsequent neighbor user.

[0120] After the server stores the mapping relationship in the cache, the server notifies the driver server, and the driver server further calls the execution server to perform statistics on the behavior data of the candidate neighbor user in the mapping relationship.

[0121] Since the execution server has completed the statistics on the behavior data of the candidate neighbor user in the mapping relationship, at this time, the behavior data of the subsequent neighbor user of the subsequent user of the subsequent block behavior data can be obtained, and the behavior data of the subsequent user and the subsequent neighbor user is replaced with the behavior data of the target user and the candidate neighbor user in the mapping relationship. The mapping relationship in the cache only stores the behavior data of the neighbor user of the user of the adjacent block behavior data, thereby saving the cache.

[0122] It should be noted that the server can obtain the behavior data of the subsequent neighbor user from an external database. In order to save the stability of the external database, the subsequent block behavior data can also be divided into subsequent sub-block behavior data, and the difference between the number of each subsequent sub-block behavior data is within a preset value range.

[0123] Each time the external database obtains the behavior data of the subsequent neighbor user of the subsequent user of a block of subsequent sub-block behavior data, until the behavior data of the subsequent neighbor user of the subsequent user of the subsequent block behavior data is obtained.

[0124] And after replacing the behavior data of the subsequent user and the subsequent neighbor user with the behavior data of the target user and the candidate neighbor user in the mapping relationship, the driver server is notified to determine the identifier of the executor in the execution server, and the executor corresponding to the identifier is used to perform statistics on the behavior data of the subsequent neighbor user in the mapping relationship based on the subsequent block behavior data.

[0125] After replacing the behavior data of the subsequent user and the subsequent neighbor user with the behavior data of the target user and the candidate neighbor user in the mapping relationship, if there are other block behavior data in the multiple block behavior data in addition to the candidate block behavior data and the subsequent block behavior data, return to execute the step of screening the adjacent subsequent block behavior data from the multiple block behavior data except the candidate block behavior data until the multiple block behavior data is screened.

[0126] If the server is a server that receives original data while dividing the original data, when the server receives the original data, S301-S305 are always executed, and when the server stops receiving the original data, S301-S305 are stopped.

[0127] Since in the Spark Streaming, the execution server generally is to count the behavior data of the adjacent users of the user of the previous block behavior data, such as, the receiving server is to receive the current block behavior data, the execution server is to count the behavior data of the adjacent users of the user of the previous block behavior data of the current block behavior data, therefore, in some embodiments, the adjacent candidate block behavior data includes the first block behavior data and the second block behavior data, and the adjacent subsequent block behavior data includes the third block behavior data and the fourth block behavior data.

[0128] Correspondingly, the method for obtaining the behavior data of the candidate adjacent user of the target user in the candidate block behavior data, generating the mapping relationship between the target user and the behavior data of the candidate adjacent user, and storing the mapping relationship in the cache, comprises:

[0129] obtaining the behavior data of the first candidate adjacent user of the first target user in the first block behavior data, and generating the first mapping relationship between the first target user and the behavior data of the first candidate adjacent user;

[0130] obtaining the behavior data of the second candidate adjacent user of the second target user in the second block behavior data, and generating the second mapping relationship between the second target user and the behavior data of the second candidate adjacent user;

[0131] storing the first mapping relationship and the second mapping relationship in the cache;

[0132] obtaining the behavior data of the subsequent adjacent user of the subsequent user of the subsequent block behavior data, and replacing the target user and the behavior data of the candidate adjacent user in the mapping relationship with the subsequent user and the behavior data of the subsequent adjacent user, comprising:

[0133] obtaining the behavior data of the first subsequent adjacent user of the first subsequent user in the third block behavior data, and obtaining the behavior data of the second subsequent adjacent user of the second subsequent user in the fourth block behavior data;

[0134] replacing the first target user and the behavior data of the first candidate adjacent user in the first mapping relationship with the first subsequent user and the behavior data of the first subsequent adjacent user;

[0135] replacing the second target user and the behavior data of the second candidate adjacent user in the second mapping relationship with the second subsequent user and the behavior data of the second subsequent adjacent user.

[0136] When the server receives the original data and divides the original data at the same time, the second block behavior data and the fourth block behavior data can be the current block behavior data, and the first block behavior data and the third block behavior data can be the previous block behavior data of the current block behavior data.

[0137] For example, the server groups the raw data received in the first second to the 200th second into first block behavior data, groups the raw data received in the 201st second to the 400th second into second block behavior data, groups the raw data received in the 401st second to the 600th second into third block behavior data, and groups the raw data received in the 601st second to the 800th second into fourth block behavior data.

[0138] In the 201st second to the 400th second, the current block behavior data is the second block behavior data, and the previous block behavior data of the current block behavior data is the first block behavior data. In the 601st second to the 800th second, the current block behavior data is the fourth block behavior data, and the previous block behavior data of the current block behavior data is the third block behavior data.

[0139] Therefore, only the current block behavior data and the previous block behavior data of the current block behavior data are stored in the cache, two-stage caching is achieved, and the cache is saved.

[0140] In addition, when the behavior data of the adjacent users of all users needs to be stored, the cache needs to be a distributed cache, and the cache has high requirements for disaster recovery, high availability, and stability. In the embodiment, since only the current block behavior data and the previous block behavior data of the current block behavior data are stored, the requirements for disaster recovery, high availability, and stability of the cache can be reduced.

[0141] In addition, when the mapping relationship is a hash table, the first mapping relationship can be a first hash table, the second mapping relationship can be a second hash table, and the behavior data of the subsequent user and the subsequent adjacent user can replace the behavior data of the target user and the candidate adjacent user in the hash table by exchanging pointers. When the mapping relationship is a state management function, the first mapping relationship and the second mapping relationship can be the same state management function, that is, the state management function simultaneously stores the behavior data of the first target user, the first candidate adjacent user, the behavior data of the second target user, and the second candidate adjacent user.

[0142] As can be seen from the above, in the embodiment of the application, the raw data is first acquired, and the raw data is divided to obtain multiple block behavior data. Then, adjacent candidate block behavior data is screened from the multiple block behavior data. Next, the behavior data of the candidate adjacent user of the target user in the candidate block behavior data is acquired, a mapping relationship between the behavior data of the target user and the candidate adjacent user is generated, and the mapping relationship is stored in the cache. Next, adjacent subsequent block behavior data other than the candidate block behavior data is screened from the multiple block behavior data. Finally, the behavior data of the subsequent adjacent user of the subsequent user of the subsequent block behavior data is acquired, and the behavior data of the subsequent user and the subsequent adjacent user replaces the behavior data of the target user and the candidate adjacent user in the mapping relationship.

[0143] That is, in the embodiment of the present application, the mapping relationship in the cache only stores the behavior data of the candidate neighboring user of the target user of the candidate block behavior data adjacent to the target user, or only stores the behavior data of the subsequent neighboring user of the subsequent user of the subsequent block behavior data adjacent to the target user, that is, the mapping relationship in the cache only stores the behavior data of the neighboring user of the user of the block behavior data adjacent to the target user, without storing the behavior data of the neighboring user of all users in the original data, thereby saving the cache occupied by the behavior data of the neighboring user.

[0144] The method described in the above embodiment will be further described in detail by way of example.

[0145] This embodiment takes the data processing device integrated in the terminal as an example, please refer to Figure 4 , Figure 4 The flowchart of the data processing method provided in the embodiment of the present application. The data processing method flow can include:

[0146] S401, the receiving server obtains the original data, and divides the original data to obtain the first block behavior data.

[0147] The original data can be the behavior data of the user under different systems collected by the backend server through the SDK. After the backend server collects the original data through the SDK, the original data can be sent to Kafka, and Kafka classifies the original data through a preset theme. The receiving server then obtains the original data in each theme of Kafka, and parses the original data in each theme to obtain a data stream. Then the receiving server divides the data stream to obtain the first block behavior data.

[0148] In addition, the receiving server can obtain the original data through the receiver in the receiving server, and divide the original data according to a preset time interval to obtain the first block behavior data.

[0149] S402, the receiving server divides the first block behavior data into each first sub-block behavior data by using a consistent hash algorithm, and the difference between the number of each first sub-block behavior data is within a preset value interval.

[0150] The process of dividing the first block behavior data into each first sub-block behavior data by using the consistent hash algorithm can be: one first sub-block behavior data corresponds to one target address on the hash ring space, the original address of each behavior data in the first block behavior data on the hash ring space is calculated, and the original address is moved clockwise, and the behavior data is distributed to the first sub-block behavior data corresponding to the first target address encountered during the clockwise movement.

[0151] S403, receiving the server obtains the behavior data of the first candidate neighboring user of the first target user of each first sub-block behavior data from the external database until the behavior data of the first candidate neighboring user of the first block behavior data of the first target user is obtained.

[0152] The first block behavior data is divided into each first sub-block behavior data, the difference between the number of each first sub-block behavior data is within a preset numerical interval, and then the target user's candidate neighboring user's behavior data of each first sub-block behavior data is obtained from the external database. Because the difference between the number of each first sub-block behavior data is within a preset numerical interval, when the behavior data of the first candidate neighboring user is queried from the external database each time, the query amount of the external database will not increase sharply in a short time, thereby ensuring the stability of the external database.

[0153] S404, the receiving server stores the identification of the first target user and the identification of the first candidate neighboring user in the first variable in the state management function, stores the identification of the first candidate neighboring user and the behavior data of the first candidate neighboring user in the second variable in the state management function, and stores the state management function in the cache.

[0154] Wherein, the receiving server can obtain the first candidate neighboring user of the first target user in the first block behavior data from the external database, and the format of the obtained first candidate neighboring user of the first target user can be (id, friendids), id represents the identification of the first target user, and friendids represents the identification of the first candidate neighboring user. The external database can be a Redis database.

[0155] The identification of the first candidate neighboring user and the behavior data of the first candidate neighboring user can be stored in the second variable in the state management function in the form of (friendids, data).

[0156] Because the integer type (int) can save cache, and the identification of the first target user and the identification of the first candidate neighboring user do not exceed uint_32, in some possible implementation manners, the data type of the identification of the target user and the identification of the candidate neighboring user can be converted to the integer type, the integer type of the identification of the target user and the integer type of the identification of the candidate neighboring user are obtained, and then the integer type of the identification of the target user and the integer type of the identification of the candidate neighboring user are stored in the first variable in the state management function, thereby saving cache.

[0157] S405, the receiving server synchronizes the first selected block behavior data to the executors in each execution server, generates storage completion information, and sends the storage completion information to the driving server to enable the driving server to determine the identifier of the executor in the execution server, and the corresponding executor of the identifier of the executor is used to count the behavior data in the state management function based on the first block behavior data.

[0158] The corresponding executor of the identifier of the executor finds the identifier of the first candidate neighboring user corresponding to the identifier of the first target user from the cached state management function based on the identifier of the target user of the first block behavior data, then finds the behavior data corresponding to the identifier of the first candidate neighboring user from the cached state management function, and then counts the behavior data corresponding to the identifier of the first candidate neighboring user, that is, performs cumulative summation on the behavior data corresponding to the identifier of the first candidate neighboring user to obtain a statistical result. Finally, the executor can periodically send the statistical result to the Hadoop Distributed File System (HDFS).

[0159] Because in Spark Streaming, the driving server is scheduled at the level of the execution server, that is, scheduled at the Node_Local level (the Node_Local level means that the behavior data and the code for calculating the behavior data are in the same execution server, but not in the same executor, that is, not in the same process, and one executor corresponds to one process), the driving server can only determine which execution server to call to count the behavior data of the first candidate neighboring user of the first target user, and cannot determine which executor in the execution server to count the behavior data of the first candidate neighboring user of the first target user. Therefore, when there are multiple executors in the execution server, the behavior data of the first candidate neighboring user of the same first target user will not be counted by the same executor, resulting in errors in the statistical result.

[0160] Therefore, in the embodiment of the application, the driving server is scheduled at the level of the executor, that is, the driving server is scheduled at the Process_Local level (the Process_Local level means that the behavior data and the code for calculating the behavior data are in the same executor, that is, in the same process), and the driving server can determine the executor that counts the behavior data of the candidate neighboring user with the identifier, so that the driving server can assign the behavior data of the neighboring user of the same user to the same executor for counting, thereby ensuring the accuracy of the statistical result.

[0161] S406, the receiving server obtains the original data and divides the original data to obtain second block behavior data.

[0162] S407, the receiving server divides the second block of behavior data into each second sub-block of behavior data by using a consistent hashing algorithm, and the difference between the number of each second sub-block of behavior data is within a preset numerical interval.

[0163] S408, the receiving server obtains the behavior data of the second candidate neighboring user of the second target user of each second sub-block of behavior data from the external database each time until the behavior data of the second candidate neighboring user of the second target user of the second block of behavior data is obtained.

[0164] S409, the receiving server stores the identity of the second target user and the identity of the second candidate neighboring user in the first variable in the state management function, stores the identity of the second candidate neighboring user and the behavior data of the second candidate neighboring user in the second variable in the state management function, and stores the state management function in the cache.

[0165] S4010, the receiving server synchronizes the second selected block of behavior data to the executor in each execution server, generates storage completion information, and sends the storage completion information to the driving server, so that the driving server determines the identity of the executor in the execution server, and the executor corresponding to the identity of the executor is used to count the behavior data in the state management function based on the second block of behavior data.

[0166] S4011, the receiving server obtains the original data and divides the original data to obtain the third block of behavior data.

[0167] S4012, the receiving server divides the third block of behavior data into each third sub-block of behavior data by using a consistent hashing algorithm, and the difference between the number of each third sub-block of behavior data is within a preset numerical interval.

[0168] S4013, the receiving server obtains the behavior data of the third candidate neighboring user of the third target user of each third sub-block of behavior data from the external database each time until the behavior data of the third candidate neighboring user of the third target user of the third block of behavior data is obtained.

[0169] S4014, the receiving server replaces the identity of the first target user and the identity of the first candidate neighboring user in the first variable of the state management function with the identity of the third target user and the identity of the third candidate neighboring user, and replaces the identity of the first candidate neighboring user and the behavior data of the first candidate neighboring user in the second variable of the state management function with the identity of the third candidate neighboring user and the behavior data of the third candidate neighboring user.

[0170] S4015, the receiving server synchronizes the third block of behavior data to the executors in each execution server, generates storage completion information, and sends the storage completion information to the driving server to enable the driving server to determine the identifier of the executor in the execution server, and the corresponding executor of the identifier of the executor is used to count the behavior data in the state management function based on the third block of behavior data.

[0171] The specific implementation process of S406~S4015 can refer to S401~S405, and this embodiment will not be described here again.

[0172] Similarly, when the receiving server divides the original data to obtain the fourth block of behavior data, the processing of the fourth block of behavior data is similar to the processing of the third block of behavior data, except that the receiving server replaces the identifier of the fourth target user and the identifier of the fourth candidate neighboring user in the second variable of the state management function with the identifier of the second target user and the identifier of the second candidate neighboring user, and replaces the identifier of the fourth candidate neighboring user and the behavior data of the fourth candidate neighboring user in the second variable of the state management function with the identifier of the second candidate neighboring user and the behavior data of the second candidate neighboring user. This embodiment will not be described here again.

[0173] In addition, the processing mode of the receiving server for the subsequently divided block of behavior data can refer to the processing mode of the third block of behavior data and the processing mode of the fourth block of behavior data. This embodiment will not be described here again.

[0174] In this embodiment, two-stage caching is realized by Spark Streaming, and only the behavior data of the first candidate neighboring user of the first target user of the first block of behavior data and the behavior data of the second candidate neighboring user of the second target user of the second block of behavior data, or only the behavior data of the second candidate neighboring user of the second target user of the second block of behavior data and the behavior data of the third candidate neighboring user of the third target user of the third block of behavior data are stored in the cache, that is, only the behavior data of the neighboring users of the users of the adjacent two blocks of behavior data is stored in the cache, without storing the behavior data of the neighboring users of all users, thereby saving the cache.

[0175] To better implement the data processing method provided in the embodiments of the application, the embodiments of the application further provide a device based on the above-mentioned data processing method. The meanings of the terms are the same as in the above-mentioned data processing method, and specific implementation details can be referred to the description in the method embodiment.

[0176] For example, as shown in Figure 5 The data processing device can include:

[0177] The first acquisition module 501 is configured to acquire original data, divide the original data, and obtain a plurality of blocks of behavior data.

[0178] The first screening module 502 is configured to screen out the adjacent candidate block behavior data from the plurality of block behavior data.

[0179] The second acquisition module 503 is configured to acquire behavior data of a candidate adjacent user of a target user in the candidate block behavior data, generate a mapping relationship between the behavior data of the target user and the candidate adjacent user, and store the mapping relationship in the cache.

[0180] The second screening module 504 is configured to screen out the adjacent subsequent block behavior data from the plurality of block behavior data, except for the candidate block behavior data.

[0181] The third acquisition module 505 is configured to acquire behavior data of a subsequent adjacent user of a subsequent user of the subsequent block behavior data, and replace the behavior data of the target user and the candidate adjacent user in the mapping relationship with the behavior data of the subsequent user and the subsequent adjacent user.

[0182] Optionally, the adjacent candidate block behavior data includes first block behavior data and second block behavior data, and the adjacent subsequent block behavior data includes third block behavior data and fourth block behavior data.

[0183] Correspondingly, the second acquisition module 503 is specifically configured to perform:

[0184] acquire behavior data of a first candidate adjacent user of a first target user in the first block behavior data, and generate a first mapping relationship between the behavior data of the first target user and the first candidate adjacent user;

[0185] acquire behavior data of a second candidate adjacent user of a second target user in the second block behavior data, and generate a second mapping relationship between the behavior data of the second target user and the second candidate adjacent user;

[0186] store the first mapping relationship and the second mapping relationship in the cache.

[0187] The third acquisition module 505 is specifically configured to perform:

[0188] acquire behavior data of a first subsequent adjacent user of a first subsequent user in the third block behavior data, and acquire behavior data of a second subsequent adjacent user of a second subsequent user in the fourth block behavior data;

[0189] replace the behavior data of the first target user and the first candidate adjacent user in the first mapping relationship with the behavior data of the first subsequent user and the first subsequent adjacent user;

[0190] replace the behavior data of the second target user and the second candidate adjacent user in the second mapping relationship with the behavior data of the second subsequent user and the second subsequent adjacent user.

[0191] Optionally, the second obtaining module 503 is specifically configured to perform:

[0192] divide the candidate block behavior data into each candidate sub-block behavior data, and a difference between quantities of each candidate sub-block behavior data is within a preset numerical interval;

[0193] obtain, each time, behavior data of a candidate neighboring user of a target user of the candidate block behavior data from the external database until the behavior data of the candidate neighboring user of the target user of the candidate block behavior data is obtained completely.

[0194] Optionally, the second obtaining module 503 is specifically configured to perform:

[0195] divide the candidate block behavior data into each candidate sub-block behavior data by using a consistent hash algorithm.

[0196] Optionally, the mapping relationship is a state management function.

[0197] Correspondingly, the second obtaining module 503 is specifically configured to perform:

[0198] obtain a candidate neighboring user of a target user in the candidate block behavior data;

[0199] store an identifier of the target user and an identifier of the candidate neighboring user in a first variable in the state management function;

[0200] obtain behavior data of the candidate neighboring user;

[0201] store the identifier of the candidate neighboring user and the behavior data of the candidate neighboring user in a second variable in the state management function;

[0202] store the state management function in a cache.

[0203] Optionally, the second obtaining module 503 is specifically configured to perform:

[0204] convert data types of the identifier of the target user and the identifier of the candidate neighboring user into integer types to obtain an integer type of the identifier of the target user and an integer type of the identifier of the candidate neighboring user;

[0205] store the integer type of the identifier of the target user and the integer type of the identifier of the candidate neighboring user in the first variable in the state management function.

[0206] Optionally, the data processing apparatus further comprises:

[0207] a synchronization module configured to synchronize the candidate block behavior data to an executor in each execution server.

[0208] The sending module is configured to generate storage completion information and send the storage completion information to the driving server, so that the driving server determines an identifier of an executor in the server, and the corresponding executor of the identifier of the executor is configured to count behavior data in the mapping relationship based on the candidate block behavior data.

[0209] In a specific implementation, each of the above modules can be implemented as an independent entity, or can be combined as the same or several entities. The specific implementation and corresponding advantages of each of the above modules can be referred to the method embodiments, and will not be described here.

[0210] Embodiments of the present application also provide an electronic device, which can be a server or a terminal, etc. Figure 6 As shown in the figure, the structure of the electronic device related to the embodiments of the present application is shown, and specifically:

[0211] The electronic device can include a processor 601 with one or more processing cores, a memory 602 with one or more computer readable storage media, a power supply 603, and an input unit 604, etc. Those skilled in the art can understand that the structure of the electronic device shown in the figure does not constitute a limitation on the electronic device, and can include more or fewer components than the figure, or combine some components, or different component arrangements. Among them: Figure 6 The processor 601 is the control center of the electronic device, which connects all parts of the electronic device through various interfaces and lines, executes computer programs and / or modules stored in the memory 602 and data stored in the memory 602, and processes various functions and data of the electronic device, so as to detect the whole electronic device. Optionally, the processor 601 can include one or more processing cores; preferably, the processor 601 can integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application program, etc., and the modem processor mainly processes wireless communication. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 601.

[0212]

[0213] ​The memory 602 can be used to store computer programs and modules, and the processor 601 executes various functions and data processing by running the computer programs and modules stored in the memory 602. The memory 602 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, computer programs required by at least one function (such as a sound playing function, an image playing function, etc.), and the like; and the data storage area can store data created according to the use of the electronic device, etc. In addition, the memory 602 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other volatile solid-state memory device. Accordingly, the memory 602 can also include a memory controller to provide the processor 601 with access to the memory 602.

[0214] The electronic device also includes a power supply 603 for powering various components. Preferably, the power supply 603 can be logically connected to the processor 601 through a power management system, so as to realize functions such as management of charging, discharging, and power consumption management through the power management system. The power supply 603 can also include one or more direct current or alternating current power supplies, a recharging system, a power supply fault detection circuit, a power supply converter or inverter, a power supply state indicator, and the like.

[0215] The electronic device can also include an input unit 604, which can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function controls.

[0216] Although not shown, the electronic device can also include a display unit and the like, which will not be described here. In particular, in the present embodiment, the processor 601 in the electronic device loads one or more executable files corresponding to the processes of the computer programs into the memory 602 according to the following instructions, and runs the computer programs stored in the memory 602 by the processor 601, so as to realize various functions, such as:

[0217] Obtaining raw data and dividing the raw data to obtain a plurality of pieces of behavior data;

[0218] Filtering out adjacent candidate piece of behavior data from the plurality of pieces of behavior data;

[0219] Obtaining behavior data of a candidate adjacent user of a target user in the candidate piece of behavior data, generating a mapping relationship between the behavior data of the target user and the candidate adjacent user, and storing the mapping relationship in a cache;

[0220] Filtering out adjacent subsequent piece of behavior data from the plurality of pieces of behavior data, except for the candidate piece of behavior data;

[0221] Obtain the behavior data of the next user's next neighbor in the subsequent block behavior data, and replace the behavior data of the target user and candidate neighbor in the mapping relationship with the behavior data of the next user and the next neighbor.

[0222] For details on the specific implementation methods and corresponding beneficial effects of the above operations, please refer to the detailed description of the data processing methods above, which will not be repeated here.

[0223] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by a computer program, or by a computer program controlling related hardware. The computer program can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0224] Therefore, embodiments of this application provide a computer-readable storage medium storing a computer program that can be loaded by a processor to execute steps in any of the data processing methods provided in embodiments of this application. For example, the computer program can execute the following steps:

[0225] Obtain the raw data and divide it into multiple behavioral data blocks;

[0226] Filter adjacent candidate blocks of behavior data from multiple blocks of behavior data;

[0227] Obtain the behavior data of the target user's candidate neighboring users from the candidate block behavior data, generate a mapping relationship between the target user and the behavior data of the candidate neighboring users, and store the mapping relationship in the cache;

[0228] Filter out adjacent subsequent block behavior data from multiple blocks of behavior data, excluding candidate block behavior data;

[0229] Obtain the behavior data of the next user's next neighbor in the subsequent block behavior data, and replace the behavior data of the target user and candidate neighbor in the mapping relationship with the behavior data of the next user and the next neighbor.

[0230] For details on the specific implementation methods and corresponding beneficial effects of the above operations, please refer to the previous embodiments, which will not be repeated here.

[0231] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0232] Due to the computer program stored in the computer readable storage medium, the steps in any of the data processing methods provided by the embodiments of the present application can be performed, thus the beneficial effects that can be achieved by any of the data processing methods provided by the embodiments of the present application can be achieved, which are described in detail in the foregoing embodiments and will not be described here again.

[0233] According to an aspect of the present application, a computer program product or computer program is provided, which includes computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device performs the above-mentioned data processing method.

[0234] The above describes in detail a data processing method, device, electronic device and computer readable storage medium provided by the embodiments of the present application. The principles and implementation manners of the present application are described by applying specific examples in this paper. The above embodiment is only used to help understand the method and core idea of the present application. Meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation manner and application range will be changed. In summary, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A data processing method, characterized in that, include: Obtain the raw data and divide it into multiple blocks of behavioral data; Filter out adjacent candidate block behavior data from the multiple blocks of behavior data; Obtain the behavior data of the candidate neighboring users of the target user in the candidate block behavior data, generate a mapping relationship between the behavior data of the target user and the candidate neighboring users, and store the mapping relationship in the cache so that the execution server can perform statistics on the behavior data of the candidate neighboring users in the mapping relationship; Filter out adjacent subsequent block behavior data from the multiple blocks of behavior data, excluding the candidate block behavior data; The behavior data of the subsequent users of the subsequent users in the subsequent block behavior data is obtained, and the behavior data of the target user and the candidate adjacent users in the mapping relationship are replaced with the behavior data of the subsequent users and the subsequent adjacent users, so that the execution server can perform statistics on the behavior data of the subsequent adjacent users in the mapping relationship based on the subsequent block behavior data.

2. The data processing method according to claim 1, characterized in that, The adjacent candidate block behavior data includes the first block behavior data and the second block behavior data, and the adjacent subsequent block behavior data includes the third block behavior data and the fourth block behavior data; Accordingly, the step of obtaining the behavior data of the candidate neighboring users of the target user in the candidate block behavior data, generating a mapping relationship between the behavior data of the target user and the candidate neighboring users, and storing the mapping relationship in a cache includes: Obtain the behavior data of the first candidate neighboring users of the first target user in the first block of behavior data, and generate a first mapping relationship between the behavior data of the first target user and the first candidate neighboring users; Obtain the behavior data of the second candidate neighboring users of the second target user from the second block of behavior data, and generate a second mapping relationship between the behavior data of the second target user and the behavior data of the second candidate neighboring users; The first mapping relationship and the second mapping relationship are stored in the cache; The step of obtaining the behavior data of the subsequent users of the subsequent user in the subsequent block behavior data, and replacing the behavior data of the target user and the candidate neighboring user in the mapping relationship with the behavior data of the subsequent user and the subsequent neighboring user, includes: Obtain the behavior data of the first subsequent neighboring user of the first subsequent user in the third block of behavior data, and obtain the behavior data of the second subsequent neighboring user of the second subsequent user in the fourth block of behavior data; Replace the behavior data of the first target user and the first candidate adjacent user in the first mapping relationship with the behavior data of the first subsequent user and the first subsequent adjacent user. Replace the behavior data of the second target user and the second candidate neighboring user in the second mapping relationship with the behavior data of the second subsequent user and the second subsequent neighboring user.

3. The data processing method according to claim 1, characterized in that, The step of obtaining the behavior data of the candidate neighboring users of the target user in the candidate block behavior data includes: The candidate block behavior data is divided into candidate sub-block behavior data, and the difference between the number of each candidate sub-block behavior data is within a preset value range. Each time, the behavior data of the candidate neighboring users of the target user are retrieved from an external database for a block of the candidate sub-block of behavior data, until the acquisition of the behavior data of the candidate neighboring users of the target user for the block of behavior data is completed.

4. The data processing method according to claim 3, characterized in that, The step of dividing the candidate block behavior data into candidate sub-block behavior data includes: The candidate block behavior data is divided into candidate sub-block behavior data using a consistent hashing algorithm.

5. The data processing method according to claim 1, characterized in that, The mapping relationship is a state management function; Accordingly, the step of obtaining the behavior data of the candidate neighboring users of the target user in the candidate block behavior data, generating a mapping relationship between the behavior data of the target user and the candidate neighboring users, and storing the mapping relationship in a cache includes: Obtain the candidate neighboring users of the target user from the candidate block behavior data; The identifier of the target user and the identifier of the candidate neighboring user are stored in the first variable of the state management function; Obtain the behavioral data of the candidate neighboring users; The identifiers and behavioral data of the candidate neighboring users are stored in the second variable of the state management function; The state management function is stored in a cache.

6. The data processing method according to claim 5, characterized in that, The step of storing the identifier of the target user and the identifier of the candidate neighboring user in the first variable of the state management function includes: Convert the data types of the target user's identifier and the candidate neighboring user's identifier to integer types to obtain the target user's identifier and the candidate neighboring user's identifier in integer type; The identifiers of the target user (integer type) and the identifiers of the candidate neighboring users (integer type) are stored in the first variable of the state management function.

7. The data processing method according to claim 1, characterized in that, After filtering out adjacent candidate block behavior data from the multiple blocks of behavior data, the method further includes: The candidate block behavior data is synchronized to the executors in each execution server; Accordingly, after storing the mapping relationship in the cache, the method further includes: A storage completion message is generated and sent to the driver server so that the driver server can determine the identifier of the executor in the execution server. The executor corresponding to the identifier is used to perform statistics on the behavior data in the mapping relationship based on the candidate block behavior data.

8. A data processing apparatus, characterized in that, include: The first acquisition module is used to acquire raw data and divide the raw data to obtain multiple blocks of behavioral data. The first filtering module is used to filter out adjacent candidate block behavior data from the multiple blocks of behavior data; The second acquisition module is used to acquire the behavior data of the candidate neighboring users of the target user in the candidate block behavior data, generate a mapping relationship between the behavior data of the target user and the candidate neighboring users, and store the mapping relationship in a cache so that the execution server can perform statistics on the behavior data of the candidate neighboring users in the mapping relationship; The second filtering module is used to filter out adjacent subsequent block behavior data from the multiple blocks of behavior data, excluding the candidate block behavior data. The third acquisition module is used to acquire the behavior data of the subsequent users of the subsequent users of the subsequent block behavior data, and replace the behavior data of the target user and the candidate adjacent user in the mapping relationship with the behavior data of the subsequent users and the subsequent adjacent users, so that the execution server can perform statistics on the behavior data of the subsequent adjacent users in the mapping relationship based on the subsequent block behavior data.

9. The data processing apparatus according to claim 8, characterized in that, The adjacent candidate block behavior data includes the first block behavior data and the second block behavior data, and the adjacent subsequent block behavior data includes the third block behavior data and the fourth block behavior data; The second acquisition module is specifically used for execution: Obtain the behavior data of the first candidate neighboring users of the first target user in the first block of behavior data, and generate a first mapping relationship between the behavior data of the first target user and the first candidate neighboring users; Obtain the behavior data of the second candidate neighboring users of the second target user from the second block of behavior data, and generate a second mapping relationship between the behavior data of the second target user and the behavior data of the second candidate neighboring users; Store the first mapping relationship and the second mapping relationship in the cache; The third acquisition module is specifically used for execution: Obtain the behavior data of the first subsequent neighboring user of the first subsequent user in the third block of behavior data, and obtain the behavior data of the second subsequent neighboring user of the second subsequent user in the fourth block of behavior data; Replace the behavior data of the first target user and the first candidate neighboring user in the first mapping relationship with the behavior data of the first subsequent user and the first subsequent adjacent user. Replace the behavior data of the second target user and the second candidate neighboring user in the second mapping relationship with the behavior data of the second subsequent user and the second subsequent neighboring user.

10. The data processing apparatus according to claim 8, characterized in that, The second acquisition module is specifically used to execute: The candidate block behavior data is divided into candidate sub-block behavior data, and the difference between the number of each candidate sub-block behavior data is within a preset value range. Each time, the behavior data of the candidate neighboring users of the target user are retrieved from an external database for a block of the candidate sub-block of behavior data, until the acquisition of the behavior data of the candidate neighboring users of the target user for the block of behavior data is completed.

11. The data processing apparatus according to claim 10, characterized in that, The second acquisition module is specifically used to execute: The candidate block behavior data is divided into candidate sub-block behavior data using a consistent hashing algorithm.

12. The data processing apparatus according to claim 8, characterized in that, The mapping relationship is a state management function; The second acquisition module is specifically used to execute: Obtain the candidate neighboring users of the target user from the candidate block behavior data; The identifier of the target user and the identifier of the candidate neighboring user are stored in the first variable of the state management function; Obtain the behavioral data of the candidate neighboring users; The identifiers and behavioral data of the candidate neighboring users are stored in the second variable of the state management function; The state management function is stored in a cache.

13. The data processing apparatus according to claim 12, characterized in that, The second acquisition module is specifically used for execution: Convert the data types of the target user's identifier and the candidate neighboring user's identifier to integer types to obtain the target user's identifier and the candidate neighboring user's identifier in integer type; The identifiers of the target user (integer type) and the identifiers of the candidate neighboring users (integer type) are stored in the first variable of the state management function.

14. The data processing apparatus according to claim 8, characterized in that, The data processing device further includes: The synchronization module is used to synchronize the candidate block behavior data to the executors in each execution server; The sending module is used to generate storage completion information and send the storage completion information to the driver server so that the driver server can determine the identifier of the executor in the execution server. The executor corresponding to the identifier of the executor is used to perform statistics on the behavior data in the mapping relationship based on the candidate block behavior data.

15. An electronic device, characterized in that, It includes a processor and a memory, the memory storing a computer program, and the processor running the computer program in the memory to perform the data processing method according to any one of claims 1 to 7.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted for loading by a processor to perform the data processing method according to any one of claims 1 to 7.

17. A computer program product, characterized in that, The computer program product stores a computer program adapted for loading by a processor to execute the data processing method according to any one of claims 1 to 7.

Images