Operational data processing method, device, computer equipment and storage medium

Abstract

The present application relates to an operation data processing method, which is applied to the field of cloud technology, and includes: obtaining operation data obtained by operating each target resource, the operation data including an object identifier and an associated resource identifier; converting each object identifier into hash data, and searching for the corresponding target index in a preset index table through each hash data; determining the address stored at the position corresponding to each target index, and distributing each object identifier and the associated resource identifier to a processing unit at the corresponding address; obtaining the corresponding object feature based on the object identifier through each processing unit, and obtaining the corresponding resource feature based on the resource identifier; splicing the object feature of each object identifier with the resource feature of the resource identifier associated with the object identifier through each processing unit to obtain target feature data; the target feature data is used to train a prediction model for predicting resource click-through rate. The use of this method can reduce the energy consumption of processing operation data and improve processing efficiency.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an operation data processing method, apparatus, computer equipment and storage medium. Background Art

[0002] With the rapid development of computer technology, computer equipment needs to run huge amounts of data at all times, such as user request data for services, browsing data, verification data and other operational data. Computer equipment implements data support for various business needs by analyzing and processing user operation data.

[0003] Traditionally, batch processing technology is used to process operational data. However, batch processing technology has high requirements on processing resources and is prone to cause congestion when processing resources are limited, resulting in low processing efficiency. Summary of the invention

[0004] Based on this, it is necessary to provide an operation data processing method, device, computer equipment and storage medium that can achieve load balancing and improve data processing efficiency in response to the above technical problems.

[0005] An operation data processing method, the method comprising:

[0006] Acquire operation data obtained by operating each target resource; the operation data includes an object identifier and an associated resource identifier;

[0007] Convert each of the object identifiers into corresponding hash data, and search for the corresponding target index in a preset index table through each of the hash data;

[0008] Determine the address stored at the location corresponding to each target index, and distribute each object identifier and the associated resource identifier to the processing unit at the corresponding address;

[0009] Acquiring corresponding object features based on the received object identifiers by each of the processing units, and acquiring corresponding resource features based on the received resource identifiers;

[0010] Through each of the processing units, the object characteristics of each of the object identifiers are spliced ​​with the resource characteristics of the resource identifier associated with the object identifier to obtain target feature data; the target feature data is used to train a prediction model, and the prediction model is used to predict the resource click rate of the resource to be processed.

[0011] An operation data processing device, the device comprising:

[0012] An identification acquisition module, used to acquire operation data obtained by operating each target resource; the operation data includes an object identification and an associated resource identification;

[0013] A conversion module, used to convert each of the object identifiers into corresponding hash data, and to search for a corresponding target index in a preset index table through each of the hash data;

[0014] A distribution module, used to determine the address stored at the position corresponding to each target index, and distribute each object identifier and the associated resource identifier to the processing unit at the corresponding address;

[0015] A feature acquisition module, configured to acquire corresponding object features based on the received object identifiers through each of the processing units, and to acquire corresponding resource features based on the received resource identifiers;

[0016] The splicing module is used to splice the object characteristics of each object identifier with the resource characteristics of the resource identifier associated with the object identifier through each processing unit to obtain target characteristic data; the target characteristic data is used to train the prediction model, and the prediction model is used to predict the resource click rate of the resource to be processed.

[0017] In one embodiment, the identification acquisition module is further used to acquire operation data obtained by operating each target resource at a preset time interval;

[0018] The device also includes a cache module; the cache module is used to obtain object identifiers and object features from the operation data, and store the object identifiers and corresponding object features in a cache space in an associated manner; obtain resource identifiers and resource features corresponding to each of the target resources from the operation data, and store each of the resource identifiers and corresponding resource features in a cache space in an associated manner.

[0019] In one embodiment, the conversion module is used to convert each of the object identifiers into corresponding hash data through a hash function, determine the conversion value corresponding to each of the object identifiers according to the length of each hash data and a preset index table, and search for the corresponding target index in the preset index table through each conversion value.

[0020] In one embodiment, the device also includes a construction module; the construction module is used to construct an empty index table and convert each candidate object identifier into corresponding candidate hash data; determine the index of each position in the empty index table according to each candidate hash data; fill the position corresponding to each index in the empty index table through the address of each processing unit to obtain a preset index table.

[0021] In one embodiment, the construction module is also used to generate random values ​​in the list of each processing unit through two random hash functions and the unit identifier of each processing unit; the number of random values ​​in the list of each processing unit is the same as the number of positions in the empty index table; based on the random values ​​in the list of each processing unit, the address of each processing unit is filled into each position in the empty index table to obtain a preset index table.

[0022] In one embodiment, the construction module is also used to determine the offset corresponding to each processing unit based on the first hash function, the unit identifier of each processing unit and the length of the empty index table; determine the jump amount corresponding to each processing unit based on the second hash function, the unit identifier of each processing unit and the length of the empty index table; the second hash function is different from the first hash function; and determine the random value in the list of each processing unit according to the offset and jump amount corresponding to each processing unit.

[0023] In one embodiment, the construction module is also used to select a target random value from the list of each processing unit, and search the empty index table for a target position corresponding to the index that is the same as the target random value; when the target position is not filled with an address, fill the target position with the address of the processing unit to which the list where the target random value is located belongs; select the next target random value from the next list of the list where the target random value is located, and return to the step of searching the empty index table for the target position corresponding to the index that is the same as the target random value and continue to execute until the filling stop condition is met, and obtain a preset index table.

[0024] In one embodiment, the construction module is also used to select the next target random value from the list of target random values ​​when the target position has been filled with an address, and return to the step of searching the empty index table for the target position corresponding to the index that is the same as the target random value and continue executing until the filling stop condition is met to obtain a preset index table.

[0025] In one embodiment, the feature acquisition module is further used to acquire object features corresponding to the received object identifier from the cache space through each of the processing units, and to acquire resource features corresponding to the received resource identifier from the cache space.

[0026] In one embodiment, the device also includes a training module; the training module is used to input each of the target feature data into a prediction model, and output the predicted click-through rate of each object identifier for the corresponding target resource; according to the difference between each predicted click-through rate and the corresponding expected click-through rate, the parameters of the prediction model are adjusted until the training stop condition is reached, thereby obtaining an updated target prediction model.

[0027] In one embodiment, the device also includes a prediction module; the prediction module is used to obtain target object features corresponding to the target object and resource features to be processed corresponding to the resource to be processed from the prediction request when a prediction request is received; the target object features and the resource features to be processed are input into the target prediction model to obtain the resource click rate output by the target prediction model; the resource click rate represents the probability of the target object clicking on the resource to be processed.

[0028] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0029] Acquire operation data obtained by operating each target resource; the operation data includes an object identifier and an associated resource identifier;

[0030] Convert each of the object identifiers into corresponding hash data, and search for the corresponding target index in a preset index table through each of the hash data;

[0031] Determine the address stored at the location corresponding to each target index, and distribute each object identifier and the associated resource identifier to the processing unit at the corresponding address;

[0032] Acquiring corresponding object features based on the received object identifiers by each of the processing units, and acquiring corresponding resource features based on the received resource identifiers;

[0033] Through each of the processing units, the object characteristics of each of the object identifiers are spliced ​​with the resource characteristics of the resource identifier associated with the object identifier to obtain target feature data; the target feature data is used to train a prediction model, and the prediction model is used to predict the resource click rate of the resource to be processed.

[0034] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0035] Acquire operation data obtained by operating each target resource; the operation data includes an object identifier and an associated resource identifier;

[0036] Convert each of the object identifiers into corresponding hash data, and search for the corresponding target index in a preset index table through each of the hash data;

[0037] Determine the address stored at the location corresponding to each target index, and distribute each object identifier and the associated resource identifier to the processing unit at the corresponding address;

[0038] Acquiring corresponding object features based on the received object identifiers by each of the processing units, and acquiring corresponding resource features based on the received resource identifiers;

[0039] Through each of the processing units, the object characteristics of each of the object identifiers are spliced ​​with the resource characteristics of the resource identifier associated with the object identifier to obtain target feature data; the target feature data is used to train a prediction model, and the prediction model is used to predict the resource click rate of the resource to be processed.

[0040] A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the following steps are implemented:

[0041] Acquire operation data obtained by operating each target resource; the operation data includes an object identifier and an associated resource identifier;

[0042] Convert each of the object identifiers into corresponding hash data, and search for the corresponding target index in a preset index table through each of the hash data;

[0043] Determine the address stored at the location corresponding to each target index, and distribute each object identifier and the associated resource identifier to the processing unit at the corresponding address;

[0044] Acquiring corresponding object features based on the received object identifiers by each of the processing units, and acquiring corresponding resource features based on the received resource identifiers;

[0045] Through each of the processing units, the object characteristics of each of the object identifiers are spliced ​​with the resource characteristics of the resource identifier associated with the object identifier to obtain target feature data; the target feature data is used to train a prediction model, and the prediction model is used to predict the resource click rate of the resource to be processed.

[0046] The above-mentioned operation data processing method, device, computer equipment, computer-readable storage medium and computer program product obtain the operation data obtained by operating each target resource, and the operation data includes an object identifier and an associated resource identifier, and each object identifier is converted into a corresponding hash data, and the corresponding target index can be accurately found in the preset index table through each hash data. The address stored at the position corresponding to each target index is determined, and each object identifier and the associated resource identifier are distributed to the processing unit at the corresponding address, so as to achieve uniform distribution of data, load balancing of the processing unit, and balanced allocation of processing resources. In addition, the distributed data is the object identifier and the associated resource identifier, the amount of distributed data is small, the transmission speed is fast, and the efficiency of data distribution can be effectively improved. Through each processing unit, the corresponding object feature is obtained based on the received object identifier, and the corresponding resource feature is obtained based on the received resource identifier, so that the object feature of each object identifier can be spliced ​​with the resource feature of the resource identifier associated with the object identifier through each processing unit, and the target feature data can be obtained, so that the processing efficiency of the operation data can be effectively improved through multiple processing units, and the data response can be timely, and the timeliness is better. The prediction model can be trained in real time through each target feature data, so that the prediction model can accurately predict the resource click rate of the resources to be processed. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 An application environment diagram of an operating data processing method in an embodiment;

[0048] Figure 2 A schematic diagram of a flow chart of an operation data processing method in one embodiment;

[0049] Figure 3 A schematic diagram of a process for obtaining operation data obtained by operating each target resource in one embodiment;

[0050] Figure 4 A schematic diagram of a flow chart of steps for constructing a preset index table in one embodiment;

[0051] Figure 5 A schematic diagram of a process of generating a random value in a list of each processing unit by using two random hash functions and a unit identifier of each processing unit in one embodiment;

[0052] Figure 6 A schematic diagram of a process of filling the address of each processing unit into each position in an empty index table based on a random value in a list of each processing unit in one embodiment to obtain a preset index table;

[0053] Figure 7 A schematic diagram of a flow chart of an operation data processing method in one of the embodiments;

[0054] Figure 8 A schematic diagram of a flow chart of an operation data processing method in another embodiment;

[0055] Fig. 9 is a structural block diagram of an operating data processing device in one embodiment;

[0056] Fig.10 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0058] The operation data processing method provided in this application can be applied to Figure 1 In the application environment shown. The present application relates to the field of artificial intelligence (AI) technology, in which artificial intelligence is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science, which attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that the machines have the functions of perception, reasoning and decision-making. The solution provided in the embodiment of the present application relates to an operation data processing method of artificial intelligence, which is specifically described by the following embodiments.

[0059] The operation data processing method provided in this application can be applied to Figure 1 The operating data processing system shown in FIG. Figure 1As shown, the operation data processing system includes a terminal 110 and a server 120, and a data storage system integrated on the server 120. The data storage system can store data that the server 120 needs to process. The data storage system can also be placed on a cloud or other network server. In one embodiment, the terminal 110 and the server 120 can each independently execute the operation data processing method provided in the embodiment of the present application. The terminal 110 and the server 120 can also be used in conjunction to execute the operation data processing method provided in the embodiment of the present application. When the terminal 110 and the server 120 are used in conjunction to execute the operation data processing method provided in the embodiment of the present application, the terminal 110 obtains the operation data obtained by operating each target resource; the operation data includes an object identifier and an associated resource identifier. The terminal 110 sends the operation data to the data storage system of the server 120. The server 120 obtains the operation data containing the object identifier and the resource identifier from the data storage system, converts each object identifier into corresponding hash data, and searches for the corresponding target index in the preset index table through each hash data. The server 120 determines the address stored at the position corresponding to each target index, and distributes each object identifier and the associated resource identifier to the processing unit at the corresponding address. The server 120 obtains the corresponding object features based on the received object identifier through each processing unit, and obtains the corresponding resource features based on the received resource identifier. The server 120 splices the object features of each object identifier with the resource features of the resource identifier associated with the object identifier through each processing unit to obtain target feature data; the target feature data is used to train the prediction model, and the prediction model is used to predict the resource click rate of the resource to be processed.

[0060] The server 120 may be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services. Cloud computing is a computing model that distributes computing tasks on a resource pool composed of a large number of computers, so that various application systems can obtain computing power, storage space and information services as needed. The network that provides resources is called a "cloud". From the user's perspective, the resources in the "cloud" can be infinitely expanded, and can be obtained at any time, used on demand, expanded at any time, and paid for by use.

[0061] The terminal 110 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, intelligent voice interaction devices, smart home appliances, vehicle-mounted terminals, and portable wearable devices. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The terminal 110 and the server 120 may be directly or indirectly connected via wired or wireless communication, which is not limited in this application.

[0062] In one embodiment, multiple servers may form a blockchain, and the servers serve as nodes on the blockchain.

[0063] In one embodiment, data related to the operation data processing method can be saved on the blockchain, such as operation data, hash data, preset index table, target index, object characteristics, resource characteristics, target characteristic data, prediction model, resources to be processed and resource click rate, etc., but not limited to this.

[0064] In one embodiment, each target resource and operation data can be obtained from a cloud social application. Cloud Social is a virtual social application model for interactive applications of the Internet of Things, cloud computing, and mobile Internet, with the purpose of establishing a famous "resource sharing relationship map" to conduct online social networking. The main feature of cloud social is to integrate and evaluate a large number of social resources to form an effective resource pool to provide services to users on demand.

[0065] It should be noted that the target resources and operation data in the relevant embodiments can be collected by the vehicle terminal in the form of Internet of Vehicles, and then processed by the operation data processing method mentioned in each embodiment of the present application to obtain target feature data. The obtained target feature data is used to train the prediction model, and the trained prediction module can predict the resource click rate of the object to be pushed or pushed to be processed (for example, advertising, audio, video, or music resources) in the Internet of Vehicles scenario.

[0066] In one embodiment, Figure 2 As shown, a method for processing operation data is provided, and the method is applied to a computer device (the computer device can be specifically Figure 1 The terminal or server in the example is used to illustrate, including the following steps:

[0067] Step S202, obtaining operation data obtained by operating each target resource; the operation data includes an object identifier and an associated resource identifier.

[0068] The target resource may be at least one of various types of information and various types of items. Various types of information include at least one of applications, texts, emoticons, pictures, audio, video, files or links, but are not limited thereto. Various types of items may include physical items and virtual items. Physical items include various physical products, which may specifically be various electronic products such as mobile phones, computers, notebooks, watches, etc., and may also be clothing products such as clothes and shoes, without making too many restrictions here.

[0069] Virtual items include but are not limited to insurance products, financial products, virtual gift resources, virtual scenes, virtual characters, virtual props, etc. Virtual scenes can specifically be game scenes in gaming devices, virtual reality simulation scenes, etc., virtual characters can specifically be various characters in the game, and virtual props can specifically be various props in the game, etc.

[0070] Operation data refers to operation data obtained by the object identifier performing a preset operation on the target resource. The object identifier may be a user identifier. The preset operation includes but is not limited to at least one of a touch operation, a voice operation, an operation through an input device such as a mouse, or a gesture operation, for example, a click operation, a double-click operation, a long press operation, a left swipe operation, or a right swipe operation, and no excessive restrictions are imposed here.

[0071] Specifically, the computer device obtains the operation data obtained by operating each target resource using the object identifier. The operation data includes the object identifier and the resource identifier associated with the object identifier. The resource identifier associated with the object identifier refers to the resource identifier corresponding to the target resource operated by the object identifier.

[0072] In one embodiment, the operation data may include multiple object identifiers and resource identifiers of multiple target resources, and the computer device may determine the resource identifiers associated with the object identifiers respectively. The multiple refers to at least two.

[0073] In one embodiment, the computer device may obtain the resource identifier and resource data obtained by operating the object identifier on each target resource, and obtain the object data corresponding to the object identifier, extract the resource feature from the resource data, and extract the object feature from the object data. The computer device may determine the resource identifier associated with the object identifier, and use the object identifier, the object feature, the resource identifier associated with the object identifier, and the resource feature of the resource identifier as the operation data corresponding to the object identifier.

[0074] Object data is data related to object identification, and may include object attribute data and object behavior data. Object attribute data is attribute data related to user objects, and may specifically include at least one of the following information: user name, gender, age, and city. Object behavior data is data related to the network behavior of user objects, and may specifically include historical click behaviors of user objects, etc.

[0075] The resource identifier may be the title, link, thumbnail, content overview, etc. of the target resource, but is not limited thereto. The resource data is data related to the resource identifier, and may be the resource type, source, content overview, all resource content, resource key content, etc. corresponding to the resource identifier, but is not limited thereto.

[0076] In one embodiment, when there are multiple object identifiers in the operation data, the computer device can determine that if the same object identifier corresponds to the same object feature, then only one copy of the object identifier and the corresponding object feature can be retained.

[0077] Step S204: convert each object identifier into corresponding hash data, and search for the corresponding target index in the preset index table through each hash data.

[0078] Among them, hash data refers to the transformation of input of arbitrary length into output of fixed length through a hash algorithm, and the output is the hash value.

[0079] In a relational database, an index is a separate, physical storage structure that sorts the values ​​of one or more columns in a database table. An index is a collection of values ​​of one or more columns in a table and a corresponding list of logical pointers to the data pages that physically identify these values ​​in the table. Each item in the index table includes an attribute value and the address of each record with the attribute value. The target index refers to the index value corresponding to the hash data in the preset index table.

[0080] The preset index table is an index table constructed in advance according to the candidate object identifiers, and the addresses of each processing unit are stored in the preset index table.

[0081] Specifically, the computer device may obtain a hash function, use each object identifier as an input of the hash function, and convert each object identifier into corresponding hash data through the hash function. The computer device may obtain a preset index table, compare each hash data with each index in the preset index table, so as to find the target index corresponding to each hash data.

[0082] Step S206, determining the address stored at the position corresponding to each target index, and distributing each object identifier and the associated resource identifier to the processing unit at the corresponding address.

[0083] Among them, the processing unit, namely Slot, is used to receive the object identifier and the resource identifier, and perform a series of processing on the object identifier and the resource identifier, for example, obtaining the object characteristics corresponding to the object identifier and the resource characteristics corresponding to the resource identifier from the cache space, and splicing the object identifier and the corresponding object characteristics, as well as the resource characteristics corresponding to the resource identifier associated with the object identifier into target characteristic data.

[0084] Specifically, after the computer device determines the target index corresponding to each hash data in the preset index table, it determines the address stored at the position corresponding to each target index, and the stored address is the address of the processing unit. The computer device distributes each object identifier and the associated resource identifier to the processing unit at the corresponding address. For example, the address stored at the position of the target index corresponding to hash data 1 obtained by converting object identifier 1 is A, and the address stored at the position of the target index corresponding to hash data 2 obtained by converting object identifier 2 is B, then object identifier 1 and the associated resource identifier are distributed to the processing unit at address A, and object identifier 2 and the associated resource identifier are distributed to the processing unit at address B.

[0085] Step S208: acquiring corresponding object features based on the received object identifier, and acquiring corresponding resource features based on the received resource identifier, through each processing unit.

[0086] Specifically, for each processing unit, after receiving the object identifier and the associated resource identifier, the object feature corresponding to the object identifier can be obtained through the object identifier, and the resource feature corresponding to the resource identifier can be obtained through the resource identifier.

[0087] In one embodiment, after receiving the object identifier and the associated resource identifier, each processing unit can obtain the object characteristics corresponding to the object identifier from the preset storage space through the object identifier, and obtain the resource characteristics corresponding to the resource identifier from the preset storage space through the resource identifier.

[0088] In one embodiment, the preset storage space may be a cache space.

[0089] Step S210, through each processing unit, the object characteristics of each object identifier and the resource characteristics of the resource identifier associated with the object identifier are spliced ​​to obtain target characteristic data; the target characteristic data is used to train the prediction model, and the prediction model is used to predict the resource click rate of the resource to be processed.

[0090] The target feature data includes an object identifier and a corresponding object feature, and a resource feature corresponding to a resource identifier associated with the object identifier.

[0091] Specifically, each processing unit can splice the object identifier, the object feature of the object identifier, and the resource feature corresponding to the resource identifier associated with the object identifier to obtain target feature data. According to the same process, the computer device can splice each object identifier and the corresponding object feature, and the corresponding resource feature into target feature data through each processing unit.

[0092] In one embodiment, the processing unit may obtain corresponding object features according to the object identifiers received within the target time period, obtain corresponding resource features according to the resource identifiers received within the target time period, and concatenate the object features of each object identifier with the resource features of the resource identifier associated with the object identifier to obtain target feature data. The target time period may be 20 minutes, 30 minutes, etc., but is not limited thereto.

[0093] In one embodiment, the computer can send each target feature data to the data set and write it in batches to a distributed file system according to a specific format required by a preset model. The distributed file system can be HDFS, which is a highly fault-tolerant and scalable distributed file system and an important part of the Hadoop system.

[0094] The target feature data can be used to train the prediction model online in real time and update the preset model. The trained prediction model or the updated prediction model can be used to predict the resource click rate of the resource to be processed. The resource click rate refers to the probability that the target object clicks on the resource to be processed.

[0095] In the above operation data processing method, the operation data obtained by operating each target resource is obtained, and the operation data includes an object identifier and an associated resource identifier. Each object identifier is converted into corresponding hash data, and the corresponding target index can be accurately found in the preset index table through each hash data. The address stored at the position corresponding to each target index is determined, and each object identifier and the associated resource identifier are distributed to the processing unit at the corresponding address, so as to achieve uniform distribution of data, load balancing of the processing unit, and balanced allocation of processing resources. In addition, the distributed data is the object identifier and the associated resource identifier, the amount of distributed data is small, the transmission speed is fast, and the efficiency of data distribution can be effectively improved. Through each processing unit, the corresponding object feature is obtained based on the received object identifier, and the corresponding resource feature is obtained based on the received resource identifier, so that the object feature of each object identifier can be spliced ​​with the resource feature of the resource identifier associated with the object identifier through each processing unit, and the target feature data can be obtained, so that the processing efficiency of the operation data can be effectively improved through multiple processing units, and the data response can be timely, and the timeliness is better. The prediction model can be trained in real time through each target feature data, so that the prediction model can accurately predict the resource click rate of the resources to be processed.

[0096] In one embodiment, Figure 3 As shown, the operation data obtained by operating each target resource includes:

[0097] Step S302: acquiring operation data obtained by operating each target resource at a preset time interval.

[0098] Specifically, the computer device can obtain the operation data obtained by operating each target resource at a preset time interval. The operation data includes an object identifier and a corresponding object feature, as well as a resource identifier associated with the object identifier and a resource feature corresponding to the resource identifier. For example, the operation data is obtained at a preset time interval of 30 minutes.

[0099] In one embodiment, the operation data includes an object identifier, object data, a resource identifier, and resource data. The computer device may extract object features from the object data and resource features from the resource data.

[0100] In one embodiment, the computer device may be set with a timer, and the timer is used to count the time. When the preset time is reached, the operation data within the preset time is obtained. It is understandable that when the preset time is reached, the timer may enter the next round of timing and accumulate the operation data in the next round of timing.

[0101] The method further includes:

[0102] Step S304: Obtain an object identifier and an object feature from the operation data, and associate the object identifier and the corresponding object feature and store them in a cache space.

[0103] Specifically, the cache space is used to temporarily store data. After the computer device obtains the object identifier and object feature from the operation data, each object identifier and the corresponding object feature are associated and stored in the cache space according to a preset format. For example, the cache space can be MapState, and the preset format can be <object identifier, object feature>.

[0104] In one embodiment, when there are multiple identical object identifiers in the operation data, and the object features corresponding to the identical object identifiers are identical, a copy of the object identifier and the corresponding object feature may be associated and stored in the cache space in a preset format.

[0105] Step S306, obtaining resource identifiers and resource features corresponding to each target resource from the operation data, and associating each resource identifier with the corresponding resource feature and storing them in the cache space.

[0106] Specifically, the computer device may determine the resource identifier corresponding to each target resource, obtain the resource feature corresponding to each resource identifier from the operation data, and then associate and store the resource identifier and the corresponding resource feature in the cache space according to a preset format. For example, the cache space may be MapState, and the preset format may be <resource identifier, resource feature>.

[0107] In this embodiment, operation data obtained by operating each target resource is obtained at preset time intervals, so as to batch process the operation data within the preset time. Object identifiers and object features are obtained from the operation data and stored in a cache space in association, resource identifiers and resource features corresponding to each target resource are obtained from the operation data, and each resource identifier and corresponding resource feature are stored in a cache space in association, so that the object identifier and object feature, resource identifier and resource feature can be temporarily stored in the cache space, so that when the data is subsequently distributed to different processing units, only the object identifier and resource identifier need to be distributed, which reduces the amount of distributed data, saves transmission resources, and can effectively improve the efficiency of data distribution.

[0108] In one embodiment, each object identifier is converted into corresponding hash data, and the corresponding target index is searched in a preset index table through each hash data, including:

[0109] Each object identifier is converted into corresponding hash data through a hash function, and the conversion value corresponding to each object identifier is determined according to the length of each hash data and the preset index table; the corresponding target index is searched in the preset index table through each conversion value.

[0110] Specifically, the computer device obtains a hash function, takes each object identifier as an input of the hash function, and converts each object identifier into corresponding hash data through the hash function. The computer device determines the length of the preset index table, which is the number of indexes in the preset index table, or the number of positions contained in the preset index table.

[0111] The computer device calculates the conversion value corresponding to the object identifier according to the hash data corresponding to the object identifier and the length of the preset index table. According to the same processing method, the conversion value corresponding to each object identifier can be obtained. Further, the hash data is a hash value.

[0112] The computer device searches for the index values ​​that are the same as the conversion value in the preset index table as the target index corresponding to the conversion value. According to the same processing method, the target index corresponding to each conversion value can be obtained.

[0113] In one embodiment, the computer device performs modulo processing on the hash data corresponding to the object identifier and the length of the preset index table to obtain the conversion value corresponding to the object identifier. Further, the remainder obtained by dividing the hash data corresponding to the object identifier by the length of the preset index table is used as the conversion value corresponding to the object identifier.

[0114] In this embodiment, each object identifier is converted into corresponding hash data through a hash function, so as to determine the conversion value corresponding to each object identifier according to the length of each hash data and the preset index table, so as to accurately and quickly find the corresponding target index in the preset index table through each conversion value, so as to determine to which processing unit each object identifier needs to be distributed for processing, thereby realizing uniform distribution of object identifiers.

[0115] In one embodiment, Figure 4 As shown, the preset index table is obtained through the construction steps, and the construction steps include:

[0116] Step S402: construct an empty index table, and convert each candidate object identifier into corresponding candidate hash data.

[0117] Specifically, the computer device can construct an empty index table of a preset length, wherein the empty index table contains a plurality of locations for storing the addresses of the processing units, and the number of the locations is the preset length of the empty index table. For example, if the empty index table contains 10 locations, the length of the empty index table is 10.

[0118] The computer device may obtain each candidate object identifier and a hash function, and use each candidate object identifier as an input of the hash function, so as to convert each candidate object identifier into corresponding candidate hash data through the hash function.

[0119] In one embodiment, the computer device may determine the number of candidate object identifiers and construct an empty index table of a preset length according to the number of candidate object identifiers. The number of candidate object identifiers is the same as the number of positions in the empty index table.

[0120] Step S404, determining the index of each position in the empty index table according to each candidate hash data.

[0121] Specifically, the computer device sets the index corresponding to each position in the empty index table according to each candidate hash data. Further, the computer device can use each candidate hash data as the index corresponding to each position in the empty index table.

[0122] In one embodiment, the computer device calculates the candidate conversion value corresponding to the candidate object identifier based on the candidate hash data corresponding to the candidate object identifier and the length of the empty index table. According to the same processing method, the candidate conversion value corresponding to each candidate object identifier can be obtained. Further, the candidate hash data is a candidate hash value. The computer device uses each candidate conversion value as an index of each position in the empty index table. It can be understood that a single candidate conversion value is used as an index of a single position.

[0123] In one embodiment, the computer device performs a modulo process on the candidate hash data corresponding to the candidate object identifier and the length of the empty index table to obtain a candidate conversion value corresponding to the candidate object identifier. Further, the remainder obtained by dividing the candidate hash data corresponding to the candidate object identifier by the length of the empty index table is used as the candidate conversion value corresponding to the candidate object identifier. Each candidate conversion value is assigned to a position in the empty index table as an index of the corresponding position.

[0124] For example, the computer device may determine the index of each position in the empty index table by the following formula:

[0125] index = hash(id)%M

[0126] Among them, index is the index, id is the object identifier, hash(id) is the hash data converted from the object identifier, and M is the length of the empty index table.

[0127] Step S406, filling the positions corresponding to the indexes in the empty index table with the addresses of the processing units to obtain a preset index table.

[0128] Specifically, the computer device obtains the address of each processing unit, and fills the address of each processing unit into each position in the empty index table based on each index in the empty index table to obtain a preset index table.

[0129] In one embodiment, the address of the processing unit is the unit index of the processing unit. The computer device fills the unit index of each processing unit into each position in the empty index table based on each index in the empty index table to obtain a preset index table.

[0130] In this embodiment, an empty index table is constructed, and each candidate object identifier is converted into corresponding candidate hash data, and the index of each position in the empty index table is determined according to each candidate hash data, so that the candidate object identifier can be mapped to each position in the empty index table in the form of hash data. The position corresponding to each index in the empty index table is filled with the address of each processing unit to save the address of each processing unit in the empty index table, so that in the subsequent operation data processing, the corresponding processing unit can be quickly found from the preset index table based on the object identifier, so that the data of the object identifier can be distributed to the corresponding processing unit for processing, so as to achieve uniform distribution of data.

[0131] In one embodiment, the positions corresponding to the indexes in the empty index table are filled by the addresses of the processing units to obtain a preset index table, including:

[0132] The random values ​​in the list of each processing unit are generated by two random hash functions and the unit identification of each processing unit; the number of random values ​​in the list of each processing unit is the same as the number of positions in the empty index table; based on the random values ​​in the list of each processing unit, the addresses of each processing unit are filled into each position in the empty index table to obtain a preset index table.

[0133] Specifically, the computer device may construct a corresponding list for each processing unit, with each processing unit corresponding to one list.

[0134] The computer device may obtain two unrelated random hash functions and determine the unit identification of each processing unit. The unit identification may be a unit name. The computer device generates a random value in the list of the processing unit through the two random hash functions and the unit identification of the processing unit. The random value in the list of each processing unit may be generated respectively in the same processing manner.

[0135] The number of random values ​​in the list of the processing unit is the same as the number of positions in the empty index table. For example, if the empty index table contains 10 positions, then the list of each processing unit contains 10 random values.

[0136] In one embodiment, the two random hash functions may be different hash functions under md5.

[0137] The computer device fills the address of the corresponding processing unit into the corresponding position of the empty index table according to the random value in each list and the index corresponding to each position in the empty index table, and obtains the preset index table after the filling is completed.

[0138] In this embodiment, two random hash functions and the unit identification of each processing unit are used to generate random values ​​in the list of each processing unit. The use of two independent and unrelated hash functions can reduce the number of collisions of the generated random values, improve the randomness of the data in the list, and ensure that the random values ​​in the list of each processing unit are evenly distributed. In addition, based on the evenly distributed random values ​​in the list of each processing unit, the addresses of each processing unit can be evenly filled into each position in the empty index table to obtain a preset index table.

[0139] In one embodiment, Figure 5 As shown, the random values ​​in the list of each processing unit are generated by two random hash functions and the unit identification of each processing unit, including:

[0140] Step S502: Determine the offset corresponding to each processing unit based on the first hash function, the unit identifier of each processing unit and the length of the empty index table.

[0141] Specifically, the computer device may obtain the first hash function, the unit identification of each processing unit, and determine the length of the empty index table. The computer device may input the unit identification of the processing unit into the first hash function for hash conversion processing, and determine the offset corresponding to the processing unit according to the value obtained by the hash conversion processing and the length of the empty index table. According to the same processing method, the offset corresponding to each processing unit can be obtained.

[0142] Furthermore, the computer device may input the unit identifier of the processing unit into the first hash function for hash conversion processing, and divide the value obtained by the hash conversion processing by the length of the empty index table to obtain a remainder, and use the remainder as the offset corresponding to the processing unit.

[0143] For example, a computer device can calculate the offset corresponding to a processing unit by the following formula:

[0144] offset = hash_1(slot_i)%M

[0145] Among them, offset is the offset, slot_i is the unit identifier of the i-th processing unit slot, hash_1 is the first hash function, and M is the length of the empty index table.

[0146] Step S504, determining the jump amount corresponding to each processing unit based on the second Hash function, the unit identifier of each processing unit and the length of the empty index table; the second Hash function is different from the first Hash function.

[0147] Specifically, the second hash function is different from the first hash function. The computer device can obtain the second hash function, the unit identification of each processing unit, and determine the length of the empty index table. The computer device can input the unit identification of the processing unit into the second hash function for hash conversion processing, and determine the jump amount corresponding to the processing unit based on the value obtained by the hash conversion processing and the length of the empty index table. According to the same processing method, the jump amount corresponding to each processing unit can be obtained.

[0148] Furthermore, the computer device may input the unit identifier of the processing unit into the second hash function for hash conversion processing, and determine the difference between the length of the empty index table and the preset value, divide the value obtained by the hash conversion processing and the difference to obtain the remainder, and use the sum of the remainder and the preset value as the jump amount corresponding to the processing unit. For example, the preset value may be 1.

[0149] For example, a computer device can calculate the jump amount corresponding to a processing unit by the following formula:

[0150] skip=hash_2(slot_i)%(M-1)+1

[0151] Among them, skipt is the skip amount, slot_i is the unit identifier of the i-th processing unit slot, hash_2 is the second hash function, and M is the length of the empty index table.

[0152] In one embodiment, the first hash function and the second hash function may be two different hash functions of the same type, for example, different hash functions under md5. The first hash function and the second hash function are independent of each other.

[0153] Step S506, determining the random value in the list of each processing unit according to the offset and jump amount corresponding to each processing unit.

[0154] Specifically, the computer device generates each random value in the list of the processing unit according to the offset and jump amount corresponding to the processing unit, and the number of random values ​​in the list is the same as the number of positions in the empty index table. According to the same processing method, each random value in the list of each processing unit can be obtained.

[0155] For example, the computer device may generate each random value in the list of processing units by the following formula:

[0156] SlotList_i[j]=(offset+j*skip)%M

[0157] Among them, SlotList_i[j] is the j-th random value in the list List of the i-th processing unit Slot, offset is the offset of the i-th processing unit Slot, skip is the jump amount of the i-th processing unit Slot, and M is the length of the empty index table.

[0158] In this embodiment, based on the first hash function, the unit identifier of each processing unit and the length of the empty index table, the offset corresponding to each processing unit is determined, and based on the second hash function, the unit identifier of each processing unit and the length of the empty index table, the jump amount corresponding to each processing unit is determined. The second hash function is different from the first hash function, and can ensure the randomness of the generated offset and jump amount. According to the offset and jump amount corresponding to each processing unit, the random value in the list of each processing unit is determined, thereby reducing the number of collisions of the generated random values, improving the randomness of the data in each list, and making the random values ​​in the list of each processing unit evenly distributed.

[0159] In one embodiment, Figure 6 As shown, based on the random value in the list of each processing unit, the address of each processing unit is filled into each position in the empty index table to obtain a preset index table, including:

[0160] Step S602, select a target random value from the list of each processing unit, and search the empty index table for a target position corresponding to the same index as the target random value.

[0161] Specifically, after generating the random values ​​in the list of each processing unit, the computer device selects a target list from each list, and selects a target random value from each random value in the target list.

[0162] In one embodiment, the computer device may select a first list from each list as the target list, and select a first random value from each random value in the target list as the target random value.

[0163] The computer device searches for the index identical to the target random value in the empty index table, and determines the target position corresponding to the index identical to the target random value.

[0164] Step S602, when the target position is not filled with an address, the address of the processing unit to which the list where the target random value is located belongs is filled into the target position.

[0165] Specifically, the computer device can determine whether the target position has been filled with the address of the processing unit. When the target position is not filled with the address, the address of the processing unit to which the list where the target random value is located belongs is obtained, and the obtained address is filled into the target position.

[0166] Step S602, select the next target random value from the next list of the list where the target random value is located, and return to the step of searching the target position corresponding to the same index as the target random value in the empty index table and continue to execute until the filling stop condition is met to obtain a preset index table.

[0167] Among them, satisfying the filling stop condition means completing the filling of the empty index table, that is, each position in the empty index table has completed the filling of the corresponding processing unit address.

[0168] Specifically, after filling the address of the processing unit to which the list where the target random value belongs to is filled into the target position, the computer device can use the next list adjacent to the list where the target random value is located as the target list. Then, the computer device searches for the target random value from each random value in the target list, and returns to the step of searching for the target position corresponding to the same index as the target random value in the empty index table and continues to execute, so as to fill the address of the corresponding processing unit in each target position where the address is not filled. Each time the address of the corresponding processing unit is filled in the target position, it can be determined whether the filling stop condition is currently met, and the filling stop condition is met to obtain a preset index table.

[0169] In one embodiment, each time the address of the corresponding processing unit is filled at the target position, it can be determined whether there are any unfilled address positions in the current index table. If there are, the filling stop condition is not met, and the next target random value is selected and the step of searching the target position corresponding to the same index as the target random value in the empty index table is returned and continued. If there are no unfilled address positions in the current index table, it means that the filling stop condition is met, and the filling of the current index table is completed to obtain the preset index table.

[0170] In this embodiment, a target random value is selected from the list of each processing unit, and the target position corresponding to the index identical to the target random value is searched in the empty index table, so that the random value of the processing unit is used as the condition for filling the data of the position in the index table. When the target position is not filled with an address, the address of the processing unit to which the list where the target random value belongs is filled into the target position, the next target random value is selected from the next list of the list where the target random value is located, and the step of searching the target position corresponding to the index identical to the target random value in the empty index table is returned and continued to be executed until the filling stop condition is met, so that the address of the processing unit can be evenly filled into the index table based on the random value of the processing unit, and it is ensured that a single position in the index table is only filled with the address of a single processing unit. The preset index table is subsequently used in the processing of the operation data, which can effectively ensure that the data is evenly distributed to different processing units for processing, realize the balanced allocation of processing resources, improve resource utilization, and improve the real-time processing efficiency of massive operation data.

[0171] In one embodiment, the method further comprises:

[0172] When the target position has been filled with an address, the next target random value is selected from the list of target random values, and the step of searching the empty index table for the target position corresponding to the same index as the target random value is returned and continued until the filling stop condition is met to obtain a preset index table.

[0173] Specifically, the computer device can determine whether the address of the processing unit has been filled at the target position. When the address of the target position has been filled, the computer device selects the next target random value from the list of target random values. After selecting the next target random value, return to the step of searching for the target position corresponding to the same index as the target random value in the empty index table and continue to execute, so as to fill the address of the corresponding processing unit at each target position that has not been filled with an address. After each time the address of the corresponding processing unit is filled at the target position, it can be determined whether the filling stop condition is currently met, and the filling stop condition is met to obtain a preset index table.

[0174] In one embodiment, each time the address of the corresponding processing unit is filled at the target position, it can be determined whether there are any unfilled address positions in the current index table. If there are, the filling stop condition is not met, and the next target random value is selected and the step of searching the target position corresponding to the same index as the target random value in the empty index table is returned and continued. If there are no unfilled address positions in the current index table, it means that the filling stop condition is met, and the filling of the current index table is completed to obtain the preset index table.

[0175] In one embodiment, when the target location is filled with addresses, the computer device selects the next random value adjacent to the target random value from the list where the target random value is located as the next target random value.

[0176] In this embodiment, when the target position has been filled with an address, the next target random value is selected from the list where the target random value is located, and the step of finding the target position corresponding to the same index as the target random value in the empty index table is returned and continued, so that the address of the processing unit can be evenly filled into the index table based on the random value of the processing unit, and it is ensured that a single position in the index table is only filled with the address of a single processing unit. The preset index table is subsequently used in the processing of operation data, which can effectively ensure that the data is evenly distributed to different processing units for processing, realize the balanced allocation of processing resources, and improve resource utilization, and can also improve the real-time processing efficiency of massive operation data.

[0177] For example, there are four processing units slot1, slot2, slot3 and slot1, and the empty index table is filled as follows:

[0178] The random value in list 1 corresponding to processing unit slot 1 is [3, 0, 4, 1, 5];

[0179] The random value in list 2 corresponding to processing unit slot 2 is [0, 2, 4, 6, 1];

[0180] The random value in list 3 corresponding to processing unit slot 3 is [3, 4, 5, 6, 0];

[0181] (1) First, traverse the first random value 3 of slot 1, and fill the address of slot 1 (i.e., the index of slot 1) into the position with index 3 (i.e., index = 3) in the index table;

[0182] (2) Then traverse the first random value 0 of slot 2 and fill the address of slot 2 into the index = 0 position in the index table;

[0183] (3) Then traverse the first random value 3 in slot 3. Because the first random value 3 in slot 3 conflicts with index = 3 in (1), that is, the position with index 3 in the index table has been filled with the address, continue to traverse the second random value 4 in slot 3. Fill the address of slot 3 into the position with index = 4 in the index table;

[0184] (4) Then traverse the second random value 0 of slot 1. Because the second random value 0 of slot 1 conflicts with index = 0 in (1), that is, the position with index 0 in the index table has been filled with the address, continue to traverse the third random value 4 of slot 1; and the third random value 4 of slot 1 conflicts with index = 4 in (3), that is, the position with index 4 in the index table has been filled with the address, continue to traverse the fourth random value 1 of slot 1, and fill the address of slot 1 into the position with index = 1 in the index table;

[0185] The process continues in this way until each position in the index table is filled, thereby obtaining a preset index table.

[0186] In one embodiment, obtaining corresponding object features based on the received object identifiers and obtaining corresponding resource features based on the received resource identifiers by each processing unit includes:

[0187] The object features corresponding to the received object identifier are acquired from the cache space by each processing unit, and the resource features corresponding to the received resource identifier are acquired from the cache space.

[0188] Specifically, after each processing unit receives the object identifier and the associated resource identifier, each processing unit obtains the object feature corresponding to the received object identifier from the cache space, and obtains the resource feature corresponding to the received resource identifier from the cache space. For example, if each processing unit receives <object identifier, resource identifier>, each processing unit can search for <object identifier, object feature> and <resource identifier, resource feature> in the cache space.

[0189] In this embodiment, the object identifier and the corresponding object feature, the resource identifier and the corresponding resource feature are stored in the cache space, and only the object identifier and the resource identifier are used to distribute to different processing units. The processing unit then obtains the corresponding feature from the cache space based on the received object identifier and resource identifier, which can reduce the amount of data distributed and avoid problems such as data distribution errors and data loss. Each processing unit can accurately obtain the corresponding feature data from the cache space for processing through the object identifier and the resource identifier, which can ensure the accuracy of the data. In addition, multiple processing units obtain the object features and resource features they need, which can achieve parallel data processing and improve data processing efficiency.

[0190] like Figure 7 FIG. 1 is a flow chart of an operation data processing method in one embodiment.

[0191] Computer equipment collects operation data under different systems through software development tool kit SDK tracking. The operation data includes each object identifier, the object data corresponding to each object identifier, and the resource identifier and resource data corresponding to each object identifier.

[0192] The computer device compresses the repeated object identifiers and corresponding object data in the operation data into one piece of data to reduce the data volume, and packages the compressed data and the associated multiple resource identifiers and resource data and reports them to the backend server, which sends the received data to Kafka. Kafka, or Apache Kafka, is an open source stream processing platform that can process all action stream data of users on the website.

[0193] The computer device can extract object features from object data and resource features from resource data, store object identifiers and corresponding object features in the format of <object identifier, object feature> in MapState, and store resource identifiers and corresponding resource features in the format of <resource identifier, resource feature> in MapState.

[0194] The computer device evenly distributes each object identifier and the associated resource identifier to the corresponding processing unit according to each object identifier, and the join operator of the processing unit associates the corresponding object features and resource features according to the received object identifier and resource identifier to obtain the target feature data. The Join operator refers to the association operator, which refers to the processing logic of the processing unit.

[0195] The processing unit sends the target feature data to the data collector and writes it in batches to the distributed file system HDFS in the specific format required by the prediction model.

[0196] In one embodiment, the computer device may generate object portraits corresponding to each object identification according to each target feature data.

[0197] In one embodiment, the method further comprises:

[0198] Input each target feature data into the prediction model, and output the predicted click rate of each object identifier for the corresponding target resource; adjust the parameters of the prediction model according to the difference between each predicted click rate and the corresponding expected click rate until the training stop condition is reached, and obtain the updated target prediction model.

[0199] Specifically, the target feature data includes an object identifier, an object feature, and a resource feature. The resource feature is a feature corresponding to the target resource, and the resource identifier of the target resource is associated with the object identifier. The computer device can input each target feature data into the prediction model, so that the prediction model outputs the predicted click rate of each object identifier for the corresponding target resource based on each target feature data. The predicted click rate refers to the predicted probability of the object identifier clicking the target resource. The computer device obtains the expected click rate, determines the difference between each predicted click rate and the expected click rate, adjusts the parameters of the prediction model according to each difference, and continues training until the training stop condition is reached, thereby obtaining a trained target prediction model.

[0200] It is understandable that, according to the above-mentioned training method, the trained prediction model can be updated in real time to obtain an updated target prediction model.

[0201] In this embodiment, the training stop condition may be at least one of the following: the loss error of the prediction model is less than or equal to the error threshold, the training iteration reaches a preset number of iterations, and the training iteration time reaches a preset iteration time.

[0202] For example, the loss error generated by the prediction model in each training is calculated, and the parameters of the prediction model are adjusted based on the difference between the loss error and the error threshold and the training is continued until the training stops, thereby obtaining a trained target prediction model or an updated target prediction model.

[0203] For example, the terminal calculates the number of iterations of the prediction model during the training process, and when the number of iterations of the training process reaches a preset number of iterations, stops the training to obtain a trained target prediction model or an updated target prediction model.

[0204] In this embodiment, the target feature data includes object identification, object features and resource features. Each target feature data is input into the prediction model, and the predicted click rate of each object identification for the corresponding target resource is output; according to the difference between each predicted click rate and the corresponding expected click rate, the parameters of the prediction model are adjusted until the training stop condition is reached, and the updated target prediction model is obtained, which can fully consider the impact of various factors on the prediction model, so that the prediction accuracy of the prediction model can be improved through training. Through the trained target prediction model or the updated target prediction model, the resource click rate of each target object for the resource to be processed can be accurately estimated.

[0205] In one embodiment, the method further comprises:

[0206] When a prediction request is received, the target object features corresponding to the target object and the to-be-processed resource features corresponding to the to-be-processed resource are obtained from the prediction request; the target object features and the to-be-processed resource features are input into the target prediction model to obtain the resource click rate output by the target prediction model; the resource click rate represents the probability of the target object clicking on the to-be-processed resource.

[0207] The resource click rate represents the probability that the target object clicks on the resource to be processed, and the resource click rate can be used to estimate the conversion rate of the resource to be processed. For example, for a certain user object, it is estimated whether the user object will click on the resource to be processed, or the probability of the user object clicking on the resource to be processed, and whether the user object will download, place an order, etc.

[0208] The prediction request is a request for estimating the resource click rate of the resource to be processed. The prediction request includes a target object, a target object feature corresponding to the target object, and a resource feature corresponding to the resource to be processed.

[0209] Specifically, when the computer device receives a prediction request for a resource click rate, the target object features corresponding to the target object and the to-be-processed resource features corresponding to the to-be-processed resource are obtained from the prediction request. The computer device inputs the target object features and the to-be-processed resource features into the target prediction model, and obtains the probability of the target object clicking the to-be-processed resource output by the target prediction model, that is, the resource click rate.

[0210] In this embodiment, when a prediction request is received, the target object features corresponding to the target object and the to-be-processed resource features corresponding to the to-be-processed resource are obtained from the prediction request. The target prediction model is used to make predictions based on the target object features and the to-be-processed resource features, so that the probability of the target object clicking the to-be-processed resource can be quickly and accurately obtained.

[0211] In one embodiment, the resource to be processed is a resource to be promoted; the method further includes: selecting a target promotion resource whose resource click rate meets the push condition from the resource to be promoted; and pushing the target promotion resource to the terminal corresponding to the corresponding target object identifier.

[0212] Specifically, the resource to be processed is a resource to be promoted. The computer device can predict the resource click rate corresponding to each resource to be promoted through the target prediction model, and select the resource to be promoted whose resource click rate is greater than the click rate threshold as the target promotion resource. The computer device pushes the target promotion resource to the terminal corresponding to the corresponding target object.

[0213] In one embodiment, the computer device can determine the resource click rate of each target object for the same resource to be promoted, and compare the click rate of each resource with the click rate threshold to determine the target number of resource click rates greater than the click rate threshold. When the target number is greater than the number threshold, the resources to be promoted corresponding to the target number are used as target promotion resources and pushed to the terminal corresponding to the target object whose resource click rate is greater than the click rate threshold.

[0214] In this embodiment, target promotion resources whose resource click-through rates meet the push conditions are selected from the resources to be promoted, and the target promotion resources are pushed to the terminal corresponding to the corresponding user object, so that the target promotion resources can be screened out based on the estimated resource click-through rates. The screened out target promotion resources have the highest conversion rate, thereby effectively improving the conversion rate of promotion resources.

[0215] In one embodiment, the method further includes: selecting a target object whose resource click rate meets the push condition based on the resources to be promoted; and pushing the corresponding resource to be promoted to the terminal corresponding to the target object.

[0216] Specifically, the computer device can determine the resource click rate of each target object for the resource to be promoted through the target prediction model, and select the target objects whose resource click rate is greater than the click rate threshold. The computer device pushes the corresponding resource to be promoted to the terminal corresponding to the selected target object.

[0217] In one embodiment, Figure 8 As shown, a method for processing operation data is provided, comprising:

[0218] Step S802: Generate a preset index table:

[0219] (1) Building an index table

[0220] Create an empty index table LookMap with a length of M. When a user ID is entered, it is mapped to the position of the index table by taking the remainder of the hash function to obtain the index of the position. The specific calculation formula is as follows:

[0221] index = hash(ID)%M

[0222] (2) Build a list of processing unit slots

[0223] For each processing unit slot, a list SlotList of length M is constructed, and the number of processing unit slots is N. The length of the list is the same as the length of the index table, which is equal to M (that is, the number of random values ​​in the list is the same as the number of indexes in the index table, which is M). The number of lists is the same as the number of processing unit slots, which is equal to N.

[0224] Generation of random values ​​in the i-th list SlotList_i:

[0225] Take two unrelated hash functions hash_1 and hash_2. The name of the i-th processing unit slot is slot_i. Use the two hash functions to calculate the offset and skip corresponding to the processing unit slot, as follows:

[0226] offset=hash_1(slot_i)%M;

[0227] skip=hash_2(slot_i)%(M-1)+1;

[0228] SlotList_i[j]=(offset+j*skip)%M

[0229] Using two independent and unrelated hash functions to reduce the number of collisions in the mapping results can ensure that the random values ​​of the lists corresponding to each slot generated are uniform and random.

[0230] (3) Fill in the index table

[0231] Initialize i, j, n: i = 0, j = 0, n = 0,

[0232] The number of filled address positions n in the index table LookMap is initialized to 0

[0233] Loop to fill the index table: while true do;

[0234] Traverse each processing unit slot, for the i-th slot: for each i <N do;

[0235] j represents the jth random value in the i-th slot list: c = SlotList_i[j]

[0236] If the position corresponding to index c in the index table has been filled with address: while LookMap[c]>=0do;

[0237] j increases by 1, and continues searching: j=j+1

[0238] c = SlotList_i[j]

[0239] End the loop until the number of unfilled address locations in the index table reaches 0: end while

[0240] Assign the address of i to the cth position in the index table: LookMap[c] = i

[0241] j=j+1

[0242] n=n+1

[0243] End the loop when all locations in the index table are filled with addresses: n = M then break

[0244] End for

[0245] End while

[0246] Step S804: Collect data and report to the server:

[0247] Through the software development tool kit SDK, object identifiers under different systems, the object data corresponding to each object identifier, and the resource identifier and resource data corresponding to each object identifier are collected.

[0248] The computer device compresses the repeated object identifiers and corresponding object data in the operation data into one piece of data, so that the data volume can be compressed to 1 / 10 of the original. The compressed data and the associated multiple resource identifiers and resource data are packaged as operation data and reported to the backend server, and the backend server sends the received data to Kafka.

[0249] It can be understood that the object identifier in this embodiment is the user id, the object data is the user data, and the object characteristics are the user characteristics; the resource identifier is the item id, the resource data is the item data, and the resource characteristics are the item characteristics.

[0250] Step S806, parsing the operation data:

[0251] Flink consumes Kafka's operational data and parses the raw data in each topic of Kafka. The operational data may include each user ID, the user data corresponding to each user ID, and the item ID and item data associated with each user ID. Flink extracts user features from user data and item features from item data. Flink is a real-time processing framework for stateful computing on streaming data.

[0252] Step S808, distribute data according to user ID:

[0253] As a result, user data is compressed. In each piece of reported data, user features and item features are in a one-to-many relationship. Therefore, Flink cannot use user ID + item ID as the key when distributing data, and cannot directly distribute it to the downstream processing unit slots based on the hash value of the user ID. Because the magnitude of user data flow is very large, and the processing unit is elastic and will be expanded frequently, resulting in changes in the number of slots, if it is directly distributed to the downstream processing unit slots based on the hash value of the user ID, the overall data distribution logic will fluctuate greatly whenever the number of slots increases or decreases, and it will cause the problem of uneven data distribution.

[0254] Therefore, in this embodiment, a custom data stream processing operator is implemented. The main logic of data stream join in the custom operator is:

[0255] The address of the corresponding processing unit slot is retrieved from the preset index table by taking the modulus of the hashed user ID, and then the user ID and the associated item ID are sent to the slot for processing in the format of <user ID, item ID>. By distributing the user ID to the corresponding processing unit, it can ensure that multiple user features and item features corresponding to the same user ID can be distributed to the same processing unit slot, which is convenient for the candidate to aggregate and count the user features and item features, and is conducive to the screening of training samples for subsequent prediction models.

[0256] Step S810, storing feature data:

[0257] Flink stores the user ID and the corresponding user features in the MapState in the format of <user ID, user features>, and stores the item ID and the corresponding item features in the MapState in the format of <item ID, item features>.

[0258] In MapState, data is stored in the form of Array[Byte].<Long,Array[Byte]> The format is fully stored according to the timestamp and is only deserialized when the feature association is finally realized, so as to reduce the serialization and deserialization overhead of reading features multiple times, while reducing the load of the recycling process GC and increasing throughput.

[0259] Serialization is the process of converting the state information of an object into a form that can be stored or transmitted. The object can be recreated by reading or deserializing the state of the object from the storage area.

[0260] Step S812: Setting a timer to associate user characteristics with item characteristics:

[0261] Register a timer in the custom operator and store the timer in the built-in rocksdb by setting Flink parameters to avoid too many timers occupying memory. The timer is used to associate user IDs, user features, and item features every 30 minutes to obtain target feature data.

[0262] Specifically, each processing unit slot obtains user features from the <user ID, user feature data> in MapState according to the user ID, and obtains item features from the <item ID, item feature data> in MapState according to the item ID. Finally, the user ID, user-side features, and item-side features are concatenated into a target feature data <user ID, user features, item features>.

[0263] Step S814: write the target feature data into the training data set:

[0264] The target feature data is sent to the data collector and written in batches to a directory in the distributed file system HDFS in the specific format required by the prediction model.

[0265] Step S816, training and updating the prediction model:

[0266] The downstream click-through rate prediction model loads the target feature data on HDFS as real-time training data, trains online, and updates the prediction model in real time. Whenever a prediction request comes in, the prediction model estimates the user's click probability on the corresponding item and passes it to the sorting module as the click-through rate factor. The sorting module sorts all items corresponding to the prediction request based on the sorting strategy and returns the top items as the result.

[0267] The operation data processing method provided in this embodiment can ensure the real-time nature of data processing, while ensuring the performance, throughput and resource utilization of real-time processing under hundreds of billions of real-time data streams and TB-level data.

[0268] In one embodiment, an operation data processing method is provided, which is applied to a computer device, comprising:

[0269] An empty index table is constructed, and each candidate object identifier is converted into a corresponding candidate hash value; and the index of each position in the empty index table is determined according to each candidate hash value and the length of the empty index table.

[0270] Next, based on the first hash function, the unit identifier of each processing unit and the length of the empty index table, the offset corresponding to each processing unit is determined; based on the second hash function, the unit identifier of each processing unit and the length of the empty index table, the jump amount corresponding to each processing unit is determined; the second hash function is different from the first hash function.

[0271] Furthermore, the random values ​​in the list of each processing unit are determined according to the offset and jump amount corresponding to each processing unit; the number of random values ​​in the list of each processing unit is the same as the number of positions in the empty index table.

[0272] Next, a target random value is selected from the list of each processing unit, and a target position corresponding to the same index as the target random value is searched in the empty index table.

[0273] Optionally, when the target position is not filled with an address, the address of the processing unit to which the list where the target random value is located belongs is filled into the target position; the next target random value is selected from the next list of the list where the target random value is located, and the step of searching the empty index table for the target position corresponding to the same index as the target random value is returned and continued.

[0274] Optionally, when the target position has been filled with an address, the next target random value is selected from the list of target random values, and the step of searching the empty index table for the target position corresponding to the same index as the target random value is returned and continued until the empty index table is filled and stopped to obtain a preset index table.

[0275] At preset time intervals, operation data obtained by operating each target resource is obtained; the operation data includes an object identifier and an associated resource identifier.

[0276] Next, the object identifier and object feature are obtained from the operation data, and the object identifier and the corresponding object feature are associated and stored in the cache space; the resource identifier and resource feature corresponding to each target resource are obtained from the operation data, and each resource identifier and the corresponding resource feature are associated and stored in the cache space.

[0277] Next, each object identifier is converted into corresponding hash data through a hash function, and the conversion value corresponding to each object identifier is determined according to the length of each hash data and the preset index table; the corresponding target index is searched in the preset index table through each conversion value.

[0278] Furthermore, the address stored at the position corresponding to each target index is determined, and each object identifier and the associated resource identifier are distributed to the processing unit at the corresponding address.

[0279] Next, each processing unit obtains the object feature corresponding to the received object identifier from the cache space, and obtains the resource feature corresponding to the received resource identifier from the cache space.

[0280] Furthermore, through each processing unit, the object feature of each object identifier is concatenated with the resource feature of the resource identifier associated with the object identifier to obtain target feature data.

[0281] Optionally, each target feature data is input into the prediction model, and the predicted click-through rate of each object identifier for the corresponding target resource is output; according to the difference between each predicted click-through rate and the corresponding expected click-through rate, the parameters of the prediction model are adjusted until the training stop condition is reached, thereby obtaining an updated target prediction model.

[0282] Optionally, when a prediction request is received, target object features corresponding to the target object and to-be-processed resource features corresponding to the to-be-processed resource are obtained from the prediction request.

[0283] Next, the target object characteristics and the to-be-processed resource characteristics are input into the target prediction model to obtain the resource click rate output by the target prediction model; the resource click rate represents the probability that the target object clicks on the to-be-processed resource.

[0284] In this embodiment, an empty index table is constructed, and each candidate object identifier is converted into corresponding candidate hash data. The index of each position in the empty index table is determined according to each candidate hash data, so that the candidate object identifier can be mapped to each position in the empty index table in the form of hash data.

[0285] The random values ​​in the list of each processing unit are generated by two random hash functions and the unit identification of each processing unit. The use of two independent and unrelated hash functions can reduce the number of collisions of the generated random values, improve the randomness of the data in the list, and ensure that the random values ​​in the list of each processing unit are evenly distributed. In addition, based on the evenly distributed random values ​​in the list of each processing unit, the addresses of each processing unit can be evenly filled into each position in the empty index table to obtain a preset index table for subsequent operation data processing.

[0286] The operation data obtained by operating each target resource is obtained at preset time intervals to batch process the operation data within the preset time. The object identifier and object feature are obtained from the operation data and stored in the cache space in association, and the resource identifier and resource feature corresponding to each target resource are obtained from the operation data, and each resource identifier and the corresponding resource feature are stored in the cache space in association. The object identifier and object feature, resource identifier and resource feature can be temporarily stored in the cache space, so that when the data is subsequently distributed to different processing units, only the object identifier and resource identifier need to be distributed, which reduces the amount of distributed data and saves transmission resources.

[0287] Each object identifier is converted into corresponding hash data through a hash function, and the conversion value corresponding to each object identifier is determined according to each hash data and the length of the preset index table, so that the corresponding target index can be accurately and quickly found in the preset index table through each conversion value, so as to determine which processing unit each object identifier needs to be distributed to for processing, and realize the uniform distribution of object identifiers and resource identifiers. And it can realize the load balancing of processing units and the balanced allocation of processing resources.

[0288] Furthermore, the data distributed is an object identifier and an associated resource identifier, the amount of data distributed is small, the transmission speed is fast, and the efficiency of data distribution can be effectively improved. Each processing unit obtains the corresponding object feature based on the received object identifier, and obtains the corresponding resource feature based on the received resource identifier, so that the object feature of each object identifier can be spliced ​​with the resource feature of the resource identifier associated with the object identifier through each processing unit to obtain the target feature data, thereby effectively improving the processing efficiency of the operation data through multiple processing units, and being able to respond to data in a timely manner, with better timeliness.

[0289] The prediction model can be trained and updated in real time through each target feature data, so that the prediction model can accurately predict the resource click rate of the resources to be processed.

[0290] It is understandable that the object information and related data involved in each embodiment are all information and data collected after authorization by the object or full authorization by all parties. Object information includes but is not limited to object device information, object personal information, such as object identification, object data, object characteristics, etc.; related data includes but is not limited to data for display, data for analysis, etc., such as target resources, resource identification and resource characteristics, etc. In addition, the object can choose not to authorize the object information and related data, or can refuse or conveniently refuse to push information, etc.

[0291] It should be understood that although Figure 2-Figure 8 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2-Figure 8 At least part of the steps may include multiple steps or multiple stages. These steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed in turn or alternately with other steps or at least part of the steps or stages in other steps.

[0292] In one embodiment, Fig. 9As shown, an operation data processing device 900 is provided. The device can adopt a software module or a hardware module, or a combination of the two to become a part of a computer device. The device specifically includes: an identification acquisition module 902, a conversion module 904, a distribution module 906, a feature acquisition module 908 and a splicing module 910, wherein:

[0293] The identification acquisition module 902 is used to acquire the operation data obtained by operating each target resource; the operation data includes the object identification and the associated resource identification.

[0294] The conversion module 904 is used to convert each object identifier into corresponding hash data, and search for the corresponding target index in the preset index table through each hash data.

[0295] The distribution module 906 is used to determine the address stored at the position corresponding to each target index, and distribute each object identifier and the associated resource identifier to the processing unit at the corresponding address.

[0296] The feature acquisition module 908 is used to acquire corresponding object features based on the received object identifier through each processing unit, and acquire corresponding resource features based on the received resource identifier.

[0297] The splicing module 910 is used to splice the object features of each object identifier with the resource features of the resource identifier associated with the object identifier through each processing unit to obtain target feature data; the target feature data is used to train a prediction model, and the prediction model is used to predict the resource click rate of the resource to be processed.

[0298] In this embodiment, the operation data obtained by operating each target resource is obtained, and the operation data includes an object identifier and an associated resource identifier. Each object identifier is converted into corresponding hash data, and the corresponding target index can be accurately found in the preset index table through each hash data. The address stored at the position corresponding to each target index is determined, and each object identifier and the associated resource identifier are distributed to the processing unit at the corresponding address, so as to achieve uniform distribution of data, load balancing of the processing unit, and balanced allocation of processing resources. In addition, the distributed data is the object identifier and the associated resource identifier, the amount of distributed data is small, the transmission speed is fast, and the efficiency of data distribution can be effectively improved. Each processing unit obtains the corresponding object feature based on the received object identifier, and obtains the corresponding resource feature based on the received resource identifier, so that the object feature of each object identifier can be spliced ​​with the resource feature of the resource identifier associated with the object identifier through each processing unit, and the target feature data can be obtained, so that the processing efficiency of the operation data can be effectively improved through multiple processing units, and the data response can be timely, and the timeliness is better. The prediction model can be trained in real time through each target feature data, so that the prediction model can accurately predict the resource click rate of the resources to be processed.

[0299] In one embodiment, the identification acquisition module 902 is further used to acquire operation data obtained by operating each target resource at a preset time interval;

[0300] The device also includes a cache module; the cache module is used to obtain object identification and object characteristics from operation data, and store the object identification and corresponding object characteristics in the cache space in association; obtain resource identification and resource characteristics corresponding to each target resource from the operation data, and store each resource identification and corresponding resource characteristic in the cache space in association.

[0301] In this embodiment, operation data obtained by operating each target resource is obtained at preset time intervals, so as to batch process the operation data within the preset time. Object identifiers and object features are obtained from the operation data and stored in a cache space in association, resource identifiers and resource features corresponding to each target resource are obtained from the operation data, and each resource identifier and corresponding resource feature are stored in a cache space in association, so that the object identifier and object feature, resource identifier and resource feature can be temporarily stored in the cache space, so that when the data is subsequently distributed to different processing units, only the object identifier and resource identifier need to be distributed, which reduces the amount of distributed data, saves transmission resources, and can effectively improve the efficiency of data distribution.

[0302] In one embodiment, the conversion module 904 is also used to convert each object identifier into corresponding hash data through a hash function, and determine the conversion value corresponding to each object identifier according to the length of each hash data and a preset index table; and search the corresponding target index in the preset index table through each conversion value.

[0303] In this embodiment, each object identifier is converted into corresponding hash data through a hash function, so as to determine the conversion value corresponding to each object identifier according to the length of each hash data and the preset index table, so as to accurately and quickly find the corresponding target index in the preset index table through each conversion value, so as to determine to which processing unit each object identifier needs to be distributed for processing, thereby realizing uniform distribution of object identifiers.

[0304] In one embodiment, the device also includes a construction module; the construction module is used to construct an empty index table and convert each candidate object identifier into corresponding candidate hash data; determine the index of each position in the empty index table according to each candidate hash data; fill the position corresponding to each index in the empty index table through the address of each processing unit to obtain a preset index table.

[0305] In this embodiment, an empty index table is constructed, and each candidate object identifier is converted into corresponding candidate hash data, and the index of each position in the empty index table is determined according to each candidate hash data, so that the candidate object identifier can be mapped to each position in the empty index table in the form of hash data. The position corresponding to each index in the empty index table is filled with the address of each processing unit to save the address of each processing unit in the empty index table, so that in the subsequent operation data processing, the corresponding processing unit can be quickly found from the preset index table based on the object identifier, so that the data of the object identifier can be distributed to the corresponding processing unit for processing, so as to achieve uniform distribution of data.

[0306] In one embodiment, the construction module is also used to generate random values ​​in the list of each processing unit through two random hash functions and the unit identifier of each processing unit; the number of random values ​​in the list of each processing unit is the same as the number of positions in the empty index table; based on the random values ​​in the list of each processing unit, the address of each processing unit is filled into each position in the empty index table to obtain a preset index table.

[0307] In this embodiment, two random hash functions and the unit identification of each processing unit are used to generate random values ​​in the list of each processing unit. The use of two independent and unrelated hash functions can reduce the number of collisions of the generated random values, improve the randomness of the data in the list, and ensure that the random values ​​in the list of each processing unit are evenly distributed. In addition, based on the evenly distributed random values ​​in the list of each processing unit, the addresses of each processing unit can be evenly filled into each position in the empty index table to obtain a preset index table.

[0308] In one embodiment, the construction module is also used to determine the offset corresponding to each processing unit based on the first hash function, the unit identifier of each processing unit and the length of the empty index table; determine the jump amount corresponding to each processing unit based on the second hash function, the unit identifier of each processing unit and the length of the empty index table; the second hash function is different from the first hash function; and determine the random value in the list of each processing unit based on the offset and jump amount corresponding to each processing unit.

[0309] In this embodiment, based on the first hash function, the unit identifier of each processing unit and the length of the empty index table, the offset corresponding to each processing unit is determined, and based on the second hash function, the unit identifier of each processing unit and the length of the empty index table, the jump amount corresponding to each processing unit is determined. The second hash function is different from the first hash function, and can ensure the randomness of the generated offset and jump amount. According to the offset and jump amount corresponding to each processing unit, the random value in the list of each processing unit is determined, thereby reducing the number of collisions of the generated random values, improving the randomness of the data in each list, and making the random values ​​in the list of each processing unit evenly distributed.

[0310] In one embodiment, the construction module is also used to select a target random value from the list of each processing unit, and search for a target position corresponding to the same index as the target random value in an empty index table; when the target position is not filled with an address, fill the target position with the address of the processing unit to which the list where the target random value is located belongs; select the next target random value from the next list of the list where the target random value is located, and return to the step of searching for the target position corresponding to the same index as the target random value in the empty index table and continue to execute until the filling stop condition is met, and obtain a preset index table.

[0311] In this embodiment, a target random value is selected from the list of each processing unit, and the target position corresponding to the index identical to the target random value is searched in the empty index table, so that the random value of the processing unit is used as the condition for filling the data of the position in the index table. When the target position is not filled with an address, the address of the processing unit to which the list where the target random value belongs is filled into the target position, the next target random value is selected from the next list of the list where the target random value is located, and the step of searching the target position corresponding to the index identical to the target random value in the empty index table is returned and continued to be executed until the filling stop condition is met, so that the address of the processing unit can be evenly filled into the index table based on the random value of the processing unit, and it is ensured that a single position in the index table is only filled with the address of a single processing unit. The preset index table is subsequently used in the processing of the operation data, which can effectively ensure that the data is evenly distributed to different processing units for processing, realize the balanced allocation of processing resources, improve resource utilization, and improve the real-time processing efficiency of massive operation data.

[0312] In one embodiment, the construction module is also used to select the next target random value from the list of target random values ​​when the target position has been filled with an address, and return to the step of searching the target position corresponding to the same index as the target random value in the empty index table and continue executing until the filling stop condition is met, thereby obtaining a preset index table.

[0313] In this embodiment, when the target position has been filled with an address, the next target random value is selected from the list where the target random value is located, and the step of finding the target position corresponding to the same index as the target random value in the empty index table is returned and continued, so that the address of the processing unit can be evenly filled into the index table based on the random value of the processing unit, and it is ensured that a single position in the index table is only filled with the address of a single processing unit. The preset index table is subsequently used in the processing of operation data, which can effectively ensure that the data is evenly distributed to different processing units for processing, realize the balanced allocation of processing resources, and improve resource utilization, and can also improve the real-time processing efficiency of massive operation data.

[0314] In one embodiment, the feature acquisition module 908 is further configured to acquire object features corresponding to the received object identifier from the cache space through each processing unit, and acquire resource features corresponding to the received resource identifier from the cache space.

[0315] In this embodiment, the object identifier and the corresponding object feature, the resource identifier and the corresponding resource feature are stored in the cache space, and only the object identifier and the resource identifier are used to distribute to different processing units. The processing unit then obtains the corresponding feature from the cache space based on the received object identifier and resource identifier, which can reduce the amount of data distributed and avoid problems such as data distribution errors and data loss. Each processing unit can accurately obtain the corresponding feature data from the cache space for processing through the object identifier and the resource identifier, which can ensure the accuracy of the data. In addition, multiple processing units obtain the object features and resource features they need, which can achieve parallel data processing and improve data processing efficiency.

[0316] In one embodiment, the device also includes a training module; the training module is used to input each target feature data into the prediction model, and output the predicted click-through rate of each object identifier for the corresponding target resource; according to the difference between each predicted click-through rate and the corresponding expected click-through rate, the parameters of the click-through rate prediction model are adjusted until the training stop condition is reached, and an updated target prediction model is obtained.

[0317] In this embodiment, the target feature data includes object identification, object features and resource features. Each target feature data is input into the prediction model, and the predicted click rate of each object identification for the corresponding target resource is output; according to the difference between each predicted click rate and the corresponding expected click rate, the parameters of the prediction model are adjusted until the training stop condition is reached, and the updated target prediction model is obtained, which can fully consider the impact of various factors on the prediction model, so that the prediction accuracy of the prediction model can be improved through training. Through the trained target prediction model or the updated target prediction model, the resource click rate of each target object for the resource to be processed can be accurately estimated.

[0318] In one embodiment, the device also includes a prediction module; the prediction module is used to obtain the target object characteristics corresponding to the target object and the to-be-processed resource characteristics corresponding to the to-be-processed resource from the prediction request when a prediction request is received; the target object characteristics and the to-be-processed resource characteristics are input into the target prediction model to obtain the resource click rate output by the target prediction model; the resource click rate represents the probability of the target object clicking the to-be-processed resource.

[0319] In this embodiment, when a prediction request is received, the target object features corresponding to the target object and the to-be-processed resource features corresponding to the to-be-processed resource are obtained from the prediction request. The target prediction model is used to make predictions based on the target object features and the to-be-processed resource features, so that the probability of the target object clicking the to-be-processed resource can be quickly and accurately obtained.

[0320] In one embodiment, the resource to be processed is a resource to be promoted; the device also includes a push module; the push module is used to select a target promotion resource whose resource click rate meets the push condition from the resource to be promoted; and push the target promotion resource to the terminal corresponding to the corresponding target object identifier.

[0321] In one embodiment, the push module is further used to select a target object whose resource click rate meets the push condition from the resources to be promoted; and push the corresponding resources to be promoted to the terminal corresponding to the target object.

[0322] For the specific definition of the operation data processing device, please refer to the definition of the operation data processing method above, which will not be repeated here. Each module in the above operation data processing device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0323] In one embodiment, a computer device is provided, which may be a server or a terminal. In this embodiment, a server is taken as an example, and its internal structure diagram may be as follows: Fig.10 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store operation data and process data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an operation data processing method is implemented.

[0324] Those skilled in the art will understand that Fig.10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0325] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0326] In one embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0327] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps in the above-mentioned method embodiments.

[0328] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0329] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0330] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for processing operation data, characterized in that: The method comprises: Acquire operation data obtained by operating each target resource; the operation data includes the user's object identifier and the associated resource identifier; Convert each of the object identifiers into corresponding hash data, and search for the corresponding target index in a preset index table through each of the hash data; Determine the address stored at the location corresponding to each target index, and distribute each object identifier and the associated resource identifier to the processing unit at the corresponding address; For each of the processing units, the targeted processing unit acquires a corresponding object feature based on the received object identifier, and the targeted processing unit acquires a corresponding resource feature based on the received resource identifier; Through the targeted processing unit, the object characteristics of the received object identifier are spliced ​​with the resource characteristics of the resource identifier associated with the object identifier to obtain the target feature data generated by the targeted processing unit; the target feature data is used to train the prediction model, and the prediction model is used to predict the resource click rate of the resource to be processed.

2. The method according to claim 1, characterized in that: The obtaining of operation data obtained by operating each target resource includes: At preset time intervals, obtain operation data obtained by operating each target resource; The method further comprises: Obtaining an object identifier and an object feature from the operation data, and associating the object identifier and the corresponding object feature and storing them in a cache space; The resource identifier and resource feature respectively corresponding to each of the target resources are obtained from the operation data, and each of the resource identifiers and the corresponding resource feature are associated and stored in the cache space.

3. The method according to claim 1, characterized in that The converting each of the object identifiers into corresponding hash data respectively, and searching for the corresponding target index in the preset index table respectively through each of the hash data, comprises: Convert each of the object identifiers into corresponding hash data using a hash function, and determine a conversion value corresponding to each of the object identifiers according to the length of each of the hash data and a preset index table; The corresponding target index is searched in the preset index table respectively through each of the conversion values.

4. The method according to claim 1, characterized in that: The preset index table is obtained through a construction step, and the construction step includes: Construct an empty index table and convert each candidate object identifier into corresponding candidate hash data; Determine the index of each position in the empty index table according to each of the candidate hash data; The positions corresponding to the indexes in the empty index table are filled with the addresses of the processing units to obtain a preset index table.

5. The method according to claim 4, characterized in that The position corresponding to each index in the empty index table is filled by the address of each processing unit to obtain a preset index table, including: Generate a random value in the list of each processing unit by using two random hash functions and the unit identifier of each processing unit; the number of random values ​​in the list of each processing unit is the same as the number of positions in the empty index table; Based on the random value in the list of each processing unit, the address of each processing unit is filled into each position in the empty index table to obtain a preset index table.

6. The method according to claim 5, characterized in that The generating of the random value in the list of each processing unit by using two random hash functions and the unit identification of each processing unit comprises: Determine the offset corresponding to each of the processing units based on the first hash function, the unit identifier of each processing unit and the length of the empty index table; Determining the jump amount corresponding to each of the processing units based on a second Hash function, a unit identifier of each processing unit and a length of the empty index table; the second Hash function is different from the first Hash function; The random value in the list of each processing unit is determined according to the offset and jump amount corresponding to each processing unit.

7. The method according to claim 5, characterized in that The address of each processing unit is filled into each position in the empty index table based on the random value in the list of each processing unit to obtain a preset index table, including: Selecting a target random value from the list of each processing unit, and searching the empty index table for a target position corresponding to the same index as the target random value; When the target position is not filled with an address, filling the target position with the address of the processing unit to which the list where the target random value is located belongs; Select the next target random value from the next list of the list where the target random value is located, and return to the step of searching the target position corresponding to the same index as the target random value in the empty index table and continue to execute until the filling stop condition is met to obtain a preset index table.

8. The method according to claim 7, characterized in that The method further comprises: When the target position has been filled with an address, the next target random value is selected from the list of target random values, and the step of searching the empty index table for the target position corresponding to the same index as the target random value is returned to and continued until the filling stop condition is met to obtain a preset index table.

9. The method according to claim 1, characterized in that: For each of the processing units, obtaining a corresponding object feature based on the received object identifier by the processing unit, and obtaining a corresponding resource feature based on the received resource identifier by the processing unit, includes: For each of the processing units, the object feature corresponding to the received object identifier is obtained from the cache space by the targeted processing unit, and the resource feature corresponding to the received resource identifier is obtained from the cache space by the targeted processing unit.

10. The method according to any one of claims 1 to 9, characterized in that: The method further comprises: Inputting each of the target feature data into a prediction model, and outputting a predicted click rate of each of the object identifiers for the corresponding target resource; According to the difference between each predicted click-through rate and the corresponding expected click-through rate, the parameters of the prediction model are adjusted until the training stop condition is reached, thereby obtaining an updated target prediction model.

11. The method according to claim 10, characterized in that The method further comprises: When receiving the prediction request, obtaining the target object feature corresponding to the target object and the to-be-processed resource feature corresponding to the to-be-processed resource from the prediction request; The target object characteristics and the to-be-processed resource characteristics are input into the target prediction model to obtain the resource click rate output by the target prediction model; the resource click rate represents the probability that the target object clicks on the to-be-processed resource.

12. An operation data processing device, characterized in that: The device comprises: An identification acquisition module is used to acquire operation data obtained by operating each target resource; the operation data includes the user's object identification and the associated resource identification; A conversion module, used to convert each of the object identifiers into corresponding hash data, and to search for a corresponding target index in a preset index table through each of the hash data; A distribution module, used to determine the address stored at the position corresponding to each target index, and distribute each object identifier and the associated resource identifier to the processing unit at the corresponding address; A feature acquisition module, configured to acquire, for each of the processing units, a corresponding object feature based on the received object identifier through the targeted processing unit, and to acquire a corresponding resource feature based on the received resource identifier through the targeted processing unit; A splicing module is used to splice the object characteristics of the received object identifier with the resource characteristics of the resource identifier associated with the object identifier through the targeted processing unit to obtain the target feature data generated by the targeted processing unit; the target feature data is used to train the prediction model, and the prediction model is used to predict the resource click rate of the resource to be processed.

13. The device according to claim 12, characterized in that The identification acquisition module is further used to acquire operation data obtained by operating each target resource at a preset time interval; The device further includes a cache module; the cache module is used to obtain an object identifier and an object feature from the operation data, and associate the object identifier and the corresponding object feature and store them in a cache space; The resource identifier and resource feature respectively corresponding to each of the target resources are obtained from the operation data, and each of the resource identifiers and the corresponding resource feature are associated and stored in the cache space.

14. The device according to claim 12, characterized in that The conversion module is used to convert each of the object identifiers into corresponding hash data through a hash function, determine the conversion value corresponding to each of the object identifiers according to the length of each of the hash data and the preset index table, and search for the corresponding target index in the preset index table through each of the conversion values.

15. The device according to claim 12, characterized in that The device also includes a construction module; the construction module is used to construct an empty index table and convert each candidate object identifier into corresponding candidate hash data; determine the index of each position in the empty index table according to each candidate hash data; The positions corresponding to the indexes in the empty index table are filled with the addresses of the processing units to obtain a preset index table.

16. The device according to claim 15, characterized in that The construction module is also used to generate random values ​​in the list of each processing unit through two random hash functions and the unit identifier of each processing unit; the number of random values ​​in the list of each processing unit is the same as the number of positions in the empty index table; based on the random values ​​in the list of each processing unit, the address of each processing unit is filled into each position in the empty index table to obtain a preset index table.

17. The device according to claim 16, characterized in that The construction module is also used to determine the offset corresponding to each processing unit based on the first hash function, the unit identifier of each processing unit and the length of the empty index table; determine the jump amount corresponding to each processing unit based on the second hash function, the unit identifier of each processing unit and the length of the empty index table; the second hash function is different from the first hash function; and determine the random value in the list of each processing unit based on the offset and jump amount corresponding to each processing unit.

18. The device according to claim 16, characterized in that The construction module is further used to select a target random value from the list of each processing unit, and search the empty index table for a target position corresponding to the same index as the target random value; when the target position is not filled with an address, fill the address of the processing unit to which the list where the target random value belongs to into the target position; Select the next target random value from the next list of the list where the target random value is located, and return to the step of searching the target position corresponding to the same index as the target random value in the empty index table and continue to execute until the filling stop condition is met to obtain a preset index table.

19. The device according to claim 18, characterized in that The construction module is also used to select the next target random value from the list of target random values ​​when the target position has been filled with an address, and return to the step of searching the target position corresponding to the same index as the target random value in the empty index table and continue to execute until the filling stop condition is met, thereby obtaining a preset index table.

20. The device according to claim 12, characterized in that The feature acquisition module is also used to obtain, for each of the processing units, object features corresponding to the received object identifier from the cache space through the targeted processing unit, and to obtain resource features corresponding to the received resource identifier from the cache space through the targeted processing unit.

21. The device according to any one of claims 12 to 20, characterized in that The device also includes a training module; the training module is used to input each of the target feature data into a prediction model, and output the predicted click-through rate of each object identifier for the corresponding target resource; according to the difference between each of the predicted click-through rates and the corresponding expected click-through rates, the parameters of the prediction model are adjusted until the training stop condition is reached, thereby obtaining an updated target prediction model.

22. The device according to claim 21, characterized in that The device also includes a prediction module; the prediction module is used to, when receiving a prediction request, obtain target object features corresponding to the target object and to-be-processed resource features corresponding to the to-be-processed resource from the prediction request; input the target object features and the to-be-processed resource features into the target prediction model to obtain a resource click rate output by the target prediction model; the resource click rate represents the probability of the target object clicking on the to-be-processed resource.

23. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 11 are implemented.

24. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

25. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

Images