User portrait construction method and device
By generating bitmap files and metadata files and recording the correspondence between user identifiers and tags, the problem of time-consuming user data query in the existing technology is solved, and the efficient construction of user portraits is achieved.
Patent Information
- Application Number
- CN202111643044.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-29
- Publication Date
- 2025-09-16
- Estimated Expiration
- 2041-12-29
AI Technical Summary
When processing massive amounts of user data, existing technologies such as Presto and SparkSQL take a long time to query user data, resulting in low efficiency in building user profiles and even inability to accurately obtain user data.
By generating bitmap files and metadata files, recording the correspondence between user IDs and user tags, storing user IDs in a bitmap format and recording location information in metadata files, user IDs can be quickly obtained to build user portraits.
It improves the efficiency of building user portraits, reduces the storage space of user identifiers, and quickly obtains user identifiers during the user portrait building process, thereby improving query efficiency.
Smart Images
Figure CN114297200B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of big data technology, and in particular to a method and device for constructing a user portrait. Background Art
[0002] With the increase in Internet users, user data such as user information and user behavior has exploded. In order to better serve users and improve user experience, it is necessary to build user portraits to analyze users.
[0003] Currently, user data is often processed using technologies such as Presto and SparkSQL to build user profiles. However, during the implementation of the present invention, the inventors discovered that the existing technology has at least the following problems: As user data increases in volume, querying user data using technologies such as Presto and SparkSQL takes a long time, or even fails to accurately retrieve user data, resulting in low efficiency in building user profiles. Summary of the Invention
[0004] In view of this, an embodiment of the present invention provides a method and device for constructing a user portrait, which can quickly obtain a user identifier corresponding to a user tag based on a bitmap file and a metadata file, thereby improving the efficiency of constructing the user portrait.
[0005] To achieve the above objectives, according to one aspect of an embodiment of the present invention, a method for constructing a user profile is provided, comprising:
[0006] Obtaining a user tag information table, wherein the user tag information table indicates a user identifier and a user tag corresponding to a user;
[0007] Based on the user tag information table, a corresponding bitmap file and a metadata file are generated; wherein the bitmap file records the user identifier in a bitmap manner, and the metadata file records the user tag and the position information of the user identifier corresponding to the user tag in the bitmap file;
[0008] When it is necessary to construct a user portrait, based on the user tag required for the user portrait, obtain the position information of the user identifier corresponding to the user tag in the bitmap file from the metadata file;
[0009] Based on the location information, obtaining a user identifier corresponding to the user tag from the bitmap file;
[0010] A user profile is constructed based on the user tag and the user identifier.
[0011] Optionally, generating a corresponding bitmap file and metadata file based on the user tag information table includes:
[0012] Setting configuration information corresponding to the user tag information table, wherein the configuration information indicates the user tag, the table name and column name of the user tag information table where the user tag is located;
[0013] The user tag information table is located according to the configuration information to generate the bitmap file and the metadata file.
[0014] Optionally, the metadata file records the user tag and the position information of the user identifier corresponding to the user tag in the bitmap file in a key-value pair manner.
[0015] Optionally, it also includes:
[0016] A modulo operation is performed on the user identifier to split the bitmap file into at least two sub-bitmap file partitions to record the user identifier.
[0017] Optionally, based on the position information, an aggregation function is used to obtain a user identifier corresponding to the user tag from the at least two sub-bitmap files.
[0018] Optionally, constructing a user profile based on the user tag and the user identifier includes:
[0019] The user tag and the user identifier are pushed to a blocking queue, so that the Spark thread continuously obtains the user tag and the user identifier from the blocking queue and constructs a user profile.
[0020] To achieve the above-mentioned purpose, according to another aspect of an embodiment of the present invention, a user portrait construction device is provided, comprising: a tag information table acquisition module, a file generation module, a location information acquisition module, a user identification acquisition module, and a user portrait construction module; wherein,
[0021] The tag information table acquisition module is used to acquire a user tag information table, wherein the user tag information table indicates a user identifier and a user tag corresponding to a user;
[0022] The file generation module is configured to generate a corresponding bitmap file and a metadata file based on the user tag information table; wherein the bitmap file records the user identifier in a bitmap format, and the metadata file records the user tag and the position information of the user identifier corresponding to the user tag in the bitmap file;
[0023] The position information acquisition module is used to acquire, from the metadata file, the position information of the user identifier corresponding to the user tag in the bitmap file based on the user tag required for the user profile when a user profile needs to be constructed;
[0024] The user identification acquisition module is used to acquire the user identification corresponding to the user tag from the bitmap file based on the location information;
[0025] The user portrait construction module is used to construct a user portrait based on the user tag and the user identifier.
[0026] Optionally, generating a corresponding bitmap file and metadata file based on the user tag information table includes:
[0027] Setting configuration information corresponding to the user tag information table, wherein the configuration information indicates the user tag, the table name and column name of the user tag information table where the user tag is located;
[0028] The user tag information table is located according to the configuration information to generate the bitmap file and the metadata file.
[0029] Optionally, the metadata file records the user tag and the position information of the user identifier corresponding to the user tag in the bitmap file in a key-value pair manner.
[0030] Optionally, the file generation module is further configured to:
[0031] A modulo operation is performed on the user identifier to split the bitmap file into at least two sub-bitmap file partitions to record the user identifier.
[0032] Optionally, the user identification acquisition module is configured to acquire the user identification corresponding to the user tag from the at least two sub-bitmap files using an aggregation function based on the location information.
[0033] Optionally, constructing a user profile based on the user tag and the user identifier includes:
[0034] The user tag and the user identifier are pushed to a blocking queue, so that the Spark thread continuously obtains the user tag and the user identifier from the blocking queue and constructs a user profile.
[0035] To achieve the above-mentioned purpose, according to another aspect of an embodiment of the present invention, an electronic device for constructing a user portrait is provided, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement any of the methods described in the above-mentioned user portrait construction method.
[0036] To achieve the above-mentioned purpose, according to another aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored. When the program is executed by a processor, any of the methods described in the above-mentioned user portrait construction method is implemented.
[0037] One embodiment of the above invention has the following advantages or beneficial effects: by pre-generating corresponding metadata files and bitmap files based on the user tag information table, the correspondence between the user tag and the user identifier is recorded, and the storage space required for the user identifier is reduced. In the process of constructing the user portrait, the location information corresponding to the user identifier can be quickly obtained from the metadata file based on the user tag, and then the corresponding user identifier can be quickly obtained from the bitmap file based on the location information, thereby improving the efficiency of constructing the user portrait.
[0038] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] The accompanying drawings are provided for a better understanding of the present invention and are not intended to limit the present invention.
[0040] Figure 1 is a schematic diagram of the main process of the user portrait construction method according to an embodiment of the present invention;
[0041] Figure 2 is a schematic diagram of a bitmap file according to an embodiment of the present invention;
[0042] Figure 3 is a schematic diagram of the correspondence between metadata files and bitmap files according to an embodiment of the present invention;
[0043] Figure 4 is a schematic diagram of the main process of another user portrait construction method according to an embodiment of the present invention;
[0044] Figure 5 is a schematic diagram of main modules of a user portrait construction device according to an embodiment of the present invention;
[0045] Figure 6 is an exemplary system architecture diagram in which embodiments of the present invention may be applied;
[0046] Figure 7 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION
[0047] The following description of exemplary embodiments of the present invention is made in conjunction with the accompanying drawings, in which various details of the embodiments of the present invention are included to facilitate understanding. These details should be considered as merely exemplary. Therefore, it should be appreciated by those skilled in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.
[0048] Figure 1 FIG. 1 is a schematic diagram of the main process of the method for constructing a user portrait according to an embodiment of the present invention. Figure 1 As shown, the user portrait construction method may specifically include the following steps:
[0049] Step S101: obtaining a user tag information table, wherein the user tag information table indicates a user identifier and a user tag corresponding to a user.
[0050] Among them, user refers to Internet user; user identification refers to any information that can be used to uniquely distinguish different users, such as user ID, user number, user account, etc. This embodiment preferentially uses user ID as an example for explanation; user tag refers to any information related to the user, which can be used to reflect the user's preferences, intentions or tendencies, etc. Depending on the actual application scenario, the user tags can be different, such as interests, hobbies, gender, age, zodiac sign, height, weight, blood type, user browsing behavior, user collection behavior, user click behavior, user purchase behavior, user login behavior, user commonly used client, user commonly used login address, user membership level, user years of use, etc. The user tag information table refers to all possible tables that record user data such as user identification and user tags. As user data increases, the user tag information table involved in the user portrait construction process may be one or multiple, as shown in Table 1 below. The user tag information table stored in the hive database is used as an example for explanation.
[0051] Table 1 Example of user tag information table
[0052] User ID gender age constellation weight blood type …… 001 male 20 Taurus 166 A …… 002 female 21 Aries 155 B …… 003 male 18 Cancer 178 AB …… …… …… …… …… …… …… ……
[0053] Step S102: Generate a corresponding bitmap file and metadata file based on the user tag information table; wherein the bitmap file records the user identifier in a bitmap format, and the metadata file records the user tag and the position information of the user identifier corresponding to the user tag in the bitmap file.
[0054] Specifically, see Figure 2The bitmap file shown uses a bitmap (or Roaring Bitmap) to record user identifiers. That is, the bytes in the storage space are sequentially numbered 001, 002, 003, etc., and 0 or 1 is written to the corresponding storage location to indicate whether the user identifier corresponding to the storage location number is stored. For example, the values written to the storage locations corresponding to numbers 001, 003, and 004 in the figure are all 1, indicating that the corresponding user identifiers 001, 003, and 004 are stored in the bitmap, while the value written to the storage location corresponding to number 002 is 0, indicating that the bitmap does not store user identifier 002. In this way, by using a bitmap to store user identifiers, the required storage space is greatly compressed, and it is convenient to quickly find whether the corresponding user identifier is stored.
[0055] It is understood that the metadata file (meta file) can record or store the correspondence between user tags and user identifiers in any form. To quickly locate or find the correspondence between user tags and user identifiers, embodiments of the present invention preferably use a custom map-like structure to store data, that is, using key-value pairs to record the location information of the user tags and the user identifiers corresponding to the user tags in the bitmap file. The specific content can be shown in the offset recorded in Table 2 below:
[0056] Table 2
[0057] Key Value User Tags Location information
[0058] Among them, the user tag is the user's gender, age, etc. The position information of the user identifier corresponding to the user tag in the bitmap file can be represented by startPos and endPos, where startPos represents the starting position and endPos represents the ending position; and the user identifier corresponding to the user tag refers to the user identifier corresponding to the user with the user tag. In addition, it is understandable that a user may have hundreds or thousands of user tags, but not every user tag needs to be used to build a user portrait. It may also be used as a screening condition in the user portrait construction process. In order to better distinguish such user tags, for each type of user tag that can be used to build a user portrait, when storing it in the metadata file, a part of content can be added to the end of the content corresponding to the user tag to indicate it. The user tag gender is used as an example to illustrate as follows:
[0059] Table 3
[0060] Key Value Male (can be used for user portraits) startPos and endPos Female (can be used for user portraits) startPos and endPos
[0061] Specifically, Figure 3Taking the correspondence between the metadata file and the bitmap file shown as an example, it can be seen that the metadata file stores multiple parts such as Offset1, Offset2, Offset3, and Offset4. Each part of the Offset records the position information of the user tag and the corresponding user identifier in the bitmap file, so as to quickly locate or find the position information of the user identifier corresponding to the user tag in the bitmap file based on the metadata file. For example, if the user tag stored in Offset1 is the gender "female", since the startPos and endPos recorded in Offset1 correspond to the storage positions 001 and 004 in the bitmap file respectively, it can be determined that the user identifier corresponding to the user tag "female" is stored in the bitmap file at positions 001-004, and the corresponding user identifiers can be obtained as 001, 002, 003, and 004, that is, the user genders corresponding to user identifiers 001, 002, 003, and 004 are all female. It can be seen that the user ID corresponding to the user tag can be quickly obtained based on the metadata file and the bitmap file, which not only saves the storage space of the user ID, but also improves the efficiency of obtaining the user ID, thereby improving the efficiency of constructing the user portrait.
[0062] In addition, it is worth noting that with the increase in internet users, the amount of user data has increased dramatically, and the number of user identifiers stored in bitmap files has also gradually increased. If the bitmap file stores too many user identifiers, such as more than 2.1 billion, the storage space may be insufficient, and the efficiency of retrieving user identifiers based on location information deserialization will be affected, which in turn affects the efficiency of building user profiles. Based on this, it is considered to split the bitmap file into at least two sub-bitmap files, or to use at least two sub-bitmap files to partition the storage of user identifiers to avoid the impact of excessive storage data on the efficiency of user identifier queries.
[0063] In an optional embodiment, a remainder operation is performed on the user identifier to split the bitmap file into at least two sub-bitmap file partitions to record the user identifier. Specifically, taking the splitting of the bitmap file into 16 sub-bitmap files D0, D1, D2, ..., D15 as an example, the user identifier can be used to perform a remainder operation on 16, so that the user identifier is stored in the corresponding sub-bitmap file according to the remainder corresponding to the user identifier. For example, if the result of the remainder operation of user identifier 001 on 16 is 1, user identifier 001 is stored in sub-bitmap file D1; and if the result of the remainder operation of user identifier 005 on 16 is 5, user identifier 005 is stored in sub-bitmap file D5. In this way, the splitting of the bitmap file can be achieved without affecting the rapid search and positioning of the user identifier in the sub-bitmap file.
[0064] On this basis, still taking the user tag information table shown in Table 1 as an example, the user tag can be obtained based on the column name in the user tag information table, and then the user ID with the corresponding user tag can be selected from the column, that is, the correspondence between the user tag and the user ID is obtained to generate metadata files and bitmap files respectively.
[0065] It is understandable that user tags and user identifiers are continuously expanded or updated over time, so the content of the user tag information table and even the number of user tag information tables are also continuously updated. In order to quickly locate the user tag information table to obtain the corresponding user tags and user identifiers, improve the efficiency of generating bitmap files and metadata files, and ensure the immediacy of bitmap files and metadata files, it is considered to define configuration information to record the user tag information table where the user tag is located, the columns in the table, and other information, and then continuously automatically generate bitmap files and metadata files based on the configuration information by triggering a scheduled task. Specifically, the configuration information corresponding to the user tag information table is set, and the configuration information indicates the user tag, the table name and column name of the user tag information table where the user tag is located; the user tag information table is located according to the configuration information to generate the bitmap file and metadata file. In addition, the configuration information can also record the location information corresponding to the column name, the type of user tag, the validity period of the user tag, the data type corresponding to the user tag, whether the user tag is used to construct the user profile, the custom delimiter or field used in the table, the total number of user tags, and any other information that may help improve the construction of the user profile.
[0066] Step S103, when it is necessary to construct a user portrait, based on the user tag required for the user portrait, obtain the position information of the user identifier corresponding to the user tag in the bitmap file from the metadata file.
[0067] Specifically, see Figure 3 , taking the user label required for the user portrait as gender "female" and the user label stored in Offset1 as gender "female" as an example, since the position information startPos and endPos recorded in Offset1 correspond to storage positions 001 and 004 in the bitmap file respectively.
[0068] Step S104: Based on the location information, obtain the user identifier corresponding to the user tag from the bitmap file.
[0069] Specifically, based on the aforementioned determination that the position information startPos and endPos correspond to storage locations 001 and 004 in the bitmap file, respectively, we can determine that the user identifier corresponding to the user tag "female" is stored at locations 001-004 in the bitmap file. Therefore, the corresponding user identifiers can be obtained as 001, 002, 003, and 004, meaning that the user genders corresponding to user identifiers 001, 002, 003, and 004 are all female. In this way, the user identifier corresponding to the user tag can be quickly obtained based on the metadata file and the bitmap file, which not only saves user identifier storage space but also improves the efficiency of obtaining user identifiers, thereby improving the efficiency of constructing user profiles.
[0070] In an optional embodiment, based on the position information, an aggregation function is used to obtain the user identifier corresponding to the user tag from the at least two sub-bitmap files. The aggregation function can be a Spark UDAF function (User Defined Aggregate Function). In other words, when a bitmap file is split into at least two sub-bitmap files for storage, if the sub-bitmap files are queried in sequence to obtain the user identifier corresponding to the user tag based on the position information, it is inevitable that the time consumed to obtain the user tag is prolonged. Therefore, it is considered to query at least two sub-bitmap files at the same time based on a custom Spark UDAF function to quickly obtain the user identifier corresponding to the user tag, thereby saving the time consumed to obtain the user tag.
[0071] Step S105: construct a user portrait based on the user tag and the user identifier.
[0072] Specifically, the Spark framework is used to calculate the number of users, user proportions and other information corresponding to each user tag to generate a user portrait. It is understandable that in order to save Spark computing resources, Spark generally starts the task thread by applying for resources first and then allocating resources. Therefore, when constructing a user portrait based on user tags and user identifiers, there will inevitably be a process of waiting for resource allocation, which will extend the time it takes to build the user portrait. Based on this, it is considered to push the user tags and user identifiers to a blocking queue so that the Spark thread can continuously obtain the user tags and user identifiers from the blocking queue and construct the user portrait. In this way, if there are user tags and user identifiers in the blocking queue, the user portrait can be constructed directly without waiting for the allocation of Spark thread computing resources, thereby improving the efficiency of mutual construction between users; and if there are no user tags and user identifiers in the blocking queue, the Spark thread can avoid problems such as wireless loops to obtain user tags and user identifiers, resulting in downtime, thereby avoiding waste of resources.
[0073] Based on the above embodiment, by pre-generating the corresponding metadata file and bitmap file based on the user tag information table, the correspondence between the user tag and the user identifier is recorded and the storage space required for the user identifier is reduced. In the process of constructing the user portrait, the location information corresponding to the user identifier can be quickly obtained from the metadata file based on the user tag, and then the corresponding user identifier can be quickly obtained from the bitmap file based on the location information, thereby improving the efficiency of constructing the user portrait.
[0074] See also Figure 4 Based on the above embodiment, the present invention provides a schematic diagram of the main process of another method for constructing a user portrait. The method may specifically include the following steps:
[0075] Step S401: Obtain a user tag information table, wherein the user tag information table indicates a user identifier and a user tag corresponding to a user.
[0076] Step S402: setting configuration information corresponding to the user tag information table, wherein the configuration information indicates the user tag, the table name and the column name of the user tag information table where the user tag is located.
[0077] Step S403: locating the user tag information table according to the configuration information to generate at least two sub-bitmap files and a metadata file.
[0078] Step S404 , when it is necessary to construct a user portrait, based on the user tag required for the user portrait, obtain the position information of the user identifier corresponding to the user tag in the sub-bitmap file from the metadata file.
[0079] Step S405 : Based on the position information, a SparkUDAF function is used to obtain a user identifier corresponding to the user tag from the at least two sub-bitmap files.
[0080] Step S406: Push the user tag and the user identifier to a blocking queue, so that the Spark thread continuously obtains the user tag and the user identifier from the blocking queue and constructs a user profile.
[0081] Based on the above embodiment, by pre-generating the corresponding metadata file and bitmap file based on the user tag information table, the correspondence between the user tag and the user identifier is recorded, and the storage space required for the user identifier is reduced. In the process of constructing the user portrait, the location information corresponding to the user identifier can be quickly obtained from the metadata file based on the user tag, and then the corresponding user identifier can be quickly obtained from the bitmap file based on the location information, thereby improving the efficiency of constructing the user portrait. In addition, by splitting the bitmap file into two or more sub-bitmap files to store the user identifier, and using an aggregate function to search synchronously, the efficiency of obtaining the user identifier is further improved. Furthermore, by pushing the user tag and user identifier to a blocking queue so that the Spark thread continuously obtains the user tag and user identifier from the blocking queue and constructs the user portrait, the waiting process of the Spark thread resource allocation is avoided, further improving the efficiency of constructing the user portrait.
[0082] See also Figure 5 Based on the above embodiment, the embodiment of the present invention provides a user portrait construction device 500, including: a tag information table acquisition module 501, a file generation module 502, a location information acquisition module 503, a user identification acquisition module 504, and a user portrait construction module 505; wherein,
[0083] The tag information table acquisition module 501 is used to acquire a user tag information table, wherein the user tag information table indicates a user identifier and a user tag corresponding to a user;
[0084] The file generation module 502 is configured to generate a corresponding bitmap file and a metadata file based on the user tag information table; wherein the bitmap file records the user identifier in a bitmap format, and the metadata file records the user tag and the position information of the user identifier corresponding to the user tag in the bitmap file;
[0085] The location information acquisition module 503 is used to acquire, from the metadata file, location information of the user identifier corresponding to the user tag in the bitmap file based on the user tag required for the user profile when a user profile needs to be constructed;
[0086] The user identification acquisition module 504 is configured to acquire the user identification corresponding to the user tag from the bitmap file based on the location information;
[0087] The user portrait building module 505 is used to build a user portrait according to the user tag and the user identifier.
[0088] In an optional implementation, generating a corresponding bitmap file and metadata file based on the user tag information table includes:
[0089] Setting configuration information corresponding to the user tag information table, wherein the configuration information indicates the user tag, the table name and column name of the user tag information table where the user tag is located;
[0090] The user tag information table is located according to the configuration information to generate the bitmap file and the metadata file.
[0091] In an optional implementation, the metadata file records the user tag and the position information of the user identifier corresponding to the user tag in the bitmap file in a key-value pair manner.
[0092] In an optional implementation, the file generation module 502 is further configured to:
[0093] A modulo operation is performed on the user identifier to split the bitmap file into at least two sub-bitmap file partitions to record the user identifier.
[0094] In an optional implementation, the user identification acquisition module 504 is configured to acquire the user identification corresponding to the user tag from the at least two sub-bitmap files using an aggregation function based on the location information.
[0095] In an optional implementation, constructing a user profile based on the user tag and the user identifier includes:
[0096] The user tag and the user identifier are pushed to a blocking queue, so that the Spark thread continuously obtains the user tag and the user identifier from the blocking queue and constructs a user profile.
[0097] Figure 6 An exemplary system architecture 600 is shown to which the user profile construction method or user profile construction apparatus according to an embodiment of the present invention may be applied.
[0098] like Figure 6 As shown, system architecture 600 may include terminal devices 601, 602, 603, a network 604, and a server 605. Network 604 is used to provide a medium for communication links between terminal devices 601, 602, 603 and server 605. Network 604 may include various connection types, such as wired or wireless communication links or fiber optic cables.
[0099] Users can use terminal devices 601, 602, and 603 to interact with server 605 via network 604 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 601, 602, and 603, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc.
[0100] The terminal devices 601 , 602 , and 603 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.
[0101] Server 605 may be a server that provides various services, such as a backend management server that supports shopping websites browsed by users using terminal devices 601, 602, and 603. The backend management server may analyze and process received data such as product information query requests, and feed back processing results, such as constructed user profiles, to the terminal devices.
[0102] It should be noted that the user portrait construction method provided in the embodiment of the present invention is generally executed by the server 605 , and accordingly, the user portrait construction device is generally set in the server 605 .
[0103] It should be understood that Figure 6 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.
[0104] Reference below Figure 7 , which shows a schematic structural diagram of a computer system 700 of a terminal device suitable for implementing an embodiment of the present invention. Figure 7 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.
[0105] like Figure 7 As shown, the computer system 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage unit 708 into a random access memory (RAM) 703. Various programs and data required for the operation of the system 700 are also stored in the RAM 703. The CPU 701, ROM 702, and RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.
[0106] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, and the like; an output section 707 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 708 including a hard disk; and a communication section 709 including a network interface card such as a LAN card or a modem. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 710 as needed, so that computer programs read therefrom can be installed into the storage section 708 as needed.
[0107] In particular, according to the embodiments disclosed in the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 709, and / or installed from a removable medium 711. When the computer program is executed by the central processing unit (CPU) 701, the above-mentioned functions defined in the system of the present invention are executed.
[0108] It should be noted that the computer-readable medium described in the present invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media can include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. This propagated data signal can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wireline, optical fiber cable, RF, or any suitable combination thereof.
[0109] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0110] The modules involved in the embodiments of the present invention may be implemented in software or hardware. The modules described may also be provided in a processor. For example, they may be described as follows: a processor including a tag information table acquisition module, a file generation module, a location information acquisition module, a user identification acquisition module, and a user portrait construction module. The names of these modules do not, in some cases, constitute a limitation on the modules themselves. For example, the user portrait construction module may also be described as a "module for constructing a user portrait based on user tags and user identifications."
[0111] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently and not be assembled into the device. The above computer-readable medium carries one or more programs. When the above one or more programs are executed by a device, the device includes: obtaining a user tag information table, wherein the user tag information table indicates the user identifier and user tag corresponding to the user; based on the user tag information table, generating a corresponding bitmap file and metadata file; wherein the bitmap file records the user identifier in a bitmap manner, and the metadata file records the user tag and the position information of the user identifier corresponding to the user tag in the bitmap file; when it is necessary to build a user portrait, based on the user tag required for the user portrait, the position information of the user identifier corresponding to the user tag in the bitmap file is obtained from the metadata file; based on the position information, the user identifier corresponding to the user tag is obtained from the bitmap file; and the user portrait is built according to the user tag and the user identifier.
[0112] According to the technical solution of an embodiment of the present invention, by pre-generating corresponding metadata files and bitmap files based on the user tag information table, the correspondence between the user tag and the user identifier is recorded and the storage space required for the user identifier is reduced. In the process of constructing the user portrait, the location information corresponding to the user identifier can be quickly obtained from the metadata file based on the user tag, and then the corresponding user identifier can be quickly obtained from the bitmap file based on the location information, thereby improving the efficiency of constructing the user portrait.
[0113] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.
Claims
1. A method for constructing a user portrait, characterized in that: include: Obtaining a user tag information table, wherein the user tag information table indicates a user identifier and a user tag corresponding to a user; Based on the user tag information table, a corresponding bitmap file and a metadata file are generated; wherein the bitmap file records the user identifier in a bitmap format, and the metadata file records the user tag and the position information of the user identifier corresponding to the user tag in the bitmap file; the metadata file records the user tag and the position information of the user identifier corresponding to the user tag in the bitmap file in a key-value pair format; the position information is represented by a starting position and an ending position; When it is necessary to construct a user portrait, based on the user tag required for the user portrait, obtain the position information of the user identifier corresponding to the user tag in the bitmap file from the metadata file; Based on the location information, obtaining a user identifier corresponding to the user tag from the bitmap file; A user profile is constructed based on the user tag and the user identifier.
2. The user portrait construction method according to claim 1, characterized in that: The generating of the corresponding bitmap file and metadata file based on the user tag information table includes: Setting configuration information corresponding to the user tag information table, wherein the configuration information indicates the user tag, the table name and column name of the user tag information table where the user tag is located; The user tag information table is located according to the configuration information to generate the bitmap file and the metadata file.
3. The user portrait construction method according to claim 1, characterized in that: Also includes: A modulo operation is performed on the user identifier to split the bitmap file into at least two sub-bitmap file partitions to record the user identifier.
4. The user portrait construction method according to claim 3, characterized in that: Based on the position information, an aggregation function is used to obtain a user identifier corresponding to the user tag from the at least two sub-bitmap files.
5. The user portrait construction method according to claim 1, characterized in that: The constructing of the user profile according to the user tag and the user identifier includes: The user tag and the user identifier are pushed to a blocking queue, so that the Spark thread continuously obtains the user tag and the user identifier from the blocking queue and constructs a user profile.
6. A user portrait construction device, characterized in that: include: Tag information table acquisition module, file generation module, location information acquisition module, user identification acquisition module, user portrait construction module; among them, The tag information table acquisition module is used to acquire a user tag information table, wherein the user tag information table indicates a user identifier and a user tag corresponding to a user; The file generation module is configured to generate a corresponding bitmap file and a metadata file based on the user tag information table; wherein the bitmap file records the user identifier in a bitmap format, and the metadata file records the position information of the user tag and the user identifier corresponding to the user tag in the bitmap file; the metadata file records the position information of the user tag and the user identifier corresponding to the user tag in the bitmap file in a key-value pair format; the position information is represented by a starting position and an ending position; The position information acquisition module is used to acquire, from the metadata file, the position information of the user identifier corresponding to the user tag in the bitmap file based on the user tag required for the user profile when a user profile needs to be constructed; The user identification acquisition module is used to acquire the user identification corresponding to the user tag from the bitmap file based on the location information; The user portrait construction module is used to construct a user portrait based on the user tag and the user identifier.
7. The user portrait construction device according to claim 6, characterized in that: The generating of the corresponding bitmap file and metadata file based on the user tag information table includes: Setting configuration information corresponding to the user tag information table, wherein the configuration information indicates the user tag, the table name and column name of the user tag information table where the user tag is located; The user tag information table is located according to the configuration information to generate the bitmap file and the metadata file.
8. The user portrait construction device according to claim 6, characterized in that: The file generation module is further used to: A modulo operation is performed on the user identifier to split the bitmap file into at least two sub-bitmap file partitions to record the user identifier.
9. The user portrait construction device according to claim 8, characterized in that: The user identification acquisition module is configured to acquire the user identification corresponding to the user tag from the at least two sub-bitmap files using an aggregation function based on the position information.
10. The user portrait construction device according to claim 6, characterized in that: The constructing of the user profile according to the user tag and the user identifier includes: The user tag and the user identifier are pushed to a blocking queue, so that the Spark thread continuously obtains the user tag and the user identifier from the blocking queue and constructs a user profile.
11. An electronic device for building a user portrait, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 5.
12. A computer-readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.
Citation Information
Patent Citations
User label storage method, user portrait calculation method and calculation equipment
CN108415978A