User portrait generation method and device based on big data and electronic equipment

Abstract

The invention discloses a user portrait generation method and device based on big data and electronic equipment, and relates to the field of data analysis, and the method comprises the following steps: 1, defining a plurality of project attributes based on a user personal data information base; step 2, after user authorization, acquiring a data acquisition authority under a specified item in a user personal data information base, performing data acquisition, and cleaning the acquired data; step 3, selecting a demand attribute of a certain application target; step 4, determining as selection, defining a keyword capturing template by taking a preset application target demand attribute as a reference, performing keyword capturing on the cleaned data by using the capturing template, and performing tagged classification; according to the method, data cleaning, keyword capturing and label mining are utilized to ensure that the used data are highly related to target requirements, an accurate user portrait is generated, and a keyword capturing template and label classification can be adjusted according to different application targets.

Description

Technical Field

[0001] The present invention relates to the field of data analysis technology, and specifically to a method, device and electronic device for generating user portraits based on big data. Background Art

[0002] User portrait is a target user model based on a series of real data. It depicts users in different dimensions, aims to mine useful information through massive user behavior data, and comprehensively display the user's information. The core of user portrait is to label users. Labels describe certain aspects of users. All labels reflecting user preference characteristics are gathered together to depict a complete user image and fully display all aspects of users. User portrait is mainly used in practical scenarios such as precision marketing, user research, personalized services and business decision-making.

[0003] However, the existing user portrait generation measures have ambiguous information descriptions due to the complex channels for obtaining user personal information, resulting in problems with the accuracy of the original data. It is difficult to effectively extract strongly correlated data, which makes the user portrait vague and contains large errors. It is also difficult to comprehensively customize the portrait goals or forms based on user needs, which limits the application audience. Therefore, there is still room for improvement. Summary of the invention

[0004] 1. Technical issues to be resolved

[0005] In view of the above-mentioned shortcomings of the prior art, the present invention provides a user portrait generation method, device and electronic device based on big data, which can effectively solve the problems of the prior art.

[0006] (II) Technical solution

[0007] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0008] The present invention discloses a method for generating a user portrait based on big data, comprising the following steps:

[0009] Step 1: Define several project attributes based on the user's personal data information database;

[0010] Step 2: After the user authorizes, the user obtains the data collection permission under the specified project in the user's personal data information database, performs data collection, and cleans the collected data;

[0011] Step 3: Select the required attributes of an application target;

[0012] Step 4: The judgment is selected. Based on the preset application target requirement attributes, a keyword crawling template is defined. The cleaned data is crawled with keywords using the crawling template, and then tagged and classified.

[0013] Step 5: Obtain the classified label data, and based on the preset application target demand attributes, use the label mining algorithm, take all labels and application target demand indicators as input, obtain all labels that are strongly correlated with the application target demand indicators, and combine them as a user profile for a certain application target;

[0014] Step 6: If it is not selected, customize the keyword crawling template, use the crawling template to crawl keywords for the cleaned data, and perform label classification;

[0015] Step 7: Get the classified label data, apply the label mining algorithm, take all labels as input, analyze the preference features, output the corresponding preference features of each category label, and combine the various preference features as the current user portrait.

[0016] Furthermore, in the process of defining the keyword crawling template in step 4, the attributes of the keyword crawling template are: defining a keyword list whose relevance meets the recognition threshold, setting context rules for keyword extraction, and searching and extracting through natural language processing technology;

[0017] Perform keyword crawling according to the defined keyword crawling template, extract keywords related to demand attributes, list the synonyms, antonyms and related words of the keywords, analyze the semantic relevance, classify according to subject, attribute, importance, etc., and establish a keyword list of related requirements.

[0018] Furthermore, the operation process of the label mining algorithm in step 5 is:

[0019] a. Extract features from the cleaned data, construct training samples, and divide them into training sets and test sets;

[0020] b. Build a label mining model through machine learning algorithms to analyze the correlation coefficient between input features and labels;

[0021] c. Predict new data based on the trained label mining model and generate labels.

[0022] Furthermore, in the process of customizing the keyword crawling template in step 6:

[0023] Actively set keywords for filtering and crawling data, determine the information crawling layout, specify crawling rules for each field, write code according to the crawling rules, crawl data and store it;

[0024] Test on different target web pages to see if the crawling effect meets expectations, and adjust the crawling rules based on the test results.

[0025] Furthermore, in the process of outputting the preference features of each category in step 7, a clustering algorithm is used to group the user tag data, calculate the similarity of each tag, obtain different representative tags, and analyze the preference feature topic categories corresponding to the representative tags.

[0026] A user portrait generation device based on big data includes:

[0027] The storage module is used to obtain the read permission of the user's personal information and to read and upload the collected and analyzed data;

[0028] A definition module is used to classify the data obtained by the storage module and define the attributes of the classified items;

[0029] The collection module is used to collect and clean data for designated projects based on the project attributes in personal information;

[0030] The preset crawling module is used to define keyword crawling templates according to preset requirements, crawl keywords from the cleaned data, and complete label classification;

[0031] The custom crawling module is used to customize the keyword crawling template. The cleaned data is crawled by keywords through the custom template and tagged and classified.

[0032] The preference analysis module is used to obtain data from the custom crawling module, use the tag mining algorithm to analyze it, output the preference characteristics of each category for all tags, and analyze the tendency of user behavior preferences in different fields;

[0033] The combination module is used to apply the label mining algorithm based on the classified label data, analyze the association with the application target demand indicators, extract the labels with strong correlation, and combine them;

[0034] The portrait generation module is used to output user portraits for a certain application target or behavioral preferences in different fields according to the trigger status of the preset crawling module and the custom crawling module, customize the storage of multi-dimensional information through graph database or document database storage, and update the user portrait regularly or in real time.

[0035] Furthermore, the combination module embeds the label data into a high-dimensional space, groups the label data using a K-means clustering algorithm, and finds label groups whose similarity to the target demand meets a threshold. The clustered label groups are defined as having high internal similarity and strong correlation with the target demand.

[0036] Furthermore, the storage module is interactively connected to the definition module and the acquisition module through a wireless network, the acquisition module is interactively connected to the preset capture module and the custom capture module through a wireless network, the custom capture module is interactively connected to the preference analysis module through a wireless network, and the combination module is interactively connected to the preset capture module, the preference analysis module and the portrait generation module through a wireless network.

[0037] An electronic device, wherein the memory stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by the processor to implement a user portrait generation method based on big data.

[0038] A computer-readable storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement a user portrait generation method based on big data.

[0039] (III) Beneficial effects

[0040] Compared with the known prior art, the technical solution provided by the present invention has the following beneficial effects:

[0041] 1. By utilizing data cleaning, keyword crawling and label mining, the acquisition of raw data has a strong directionality, ensuring that the data used are highly relevant to the target needs, generating accurate user portraits, and then accurately reflecting user behaviors, interests and needs. The keyword crawling templates and label classifications can be adjusted according to different application goals, and then adapt to a variety of business needs, thereby supporting more accurate personalized recommendations, marketing and services.

[0042] 2. Users can choose preset application target demand attributes or customize keyword capture templates, which not only makes the user portrait adapt to the adjustment of application targets, but also generates user portraits that can be used as business recommendations by analyzing user preferences when there are no application targets, making user portrait generation highly personalized. Ultimately, user portraits can adapt to different application scenarios and user preferences, providing customized solutions for the specific needs of different user groups.

[0043] 3. In the tag mining and combination stage, the algorithm automatically processes large amounts of data and quickly identifies the tags most relevant to the target needs, which greatly improves the efficiency of user portrait generation. It is able to process large amounts of user data in a short period of time and generate high-quality portraits to adapt to the rapidly changing market environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0045] Figure 1 It is a flowchart of the method for generating user portraits based on big data in the present invention;

[0046] Figure 2 It is a schematic diagram of the framework of the user portrait generation device based on big data in the present invention.

[0047] The numbers in the figure represent: 1. Storage module; 2. Definition module; 3. Collection module; 4. Preset capture module; 5. Custom capture module; 6. Preference analysis module; 7. Combination module; 8. Portrait generation module. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0049] The present invention will be further described below in conjunction with the embodiments.

[0050] Example

[0051] The user portrait generation method based on big data in this embodiment is as follows: Figure 1 As shown, the following steps are included:

[0052] Step 1: Define several project attributes based on the user's personal data information database. First, obtain and organize the data in the user's personal data information database, including the user's basic information, behavior records and purchase history. Based on the above data, define several attributes related to the target application, including interests, hobbies, purchase tendencies and social behaviors, to ensure that subsequent data collection and analysis are closely centered around such attributes, thereby improving the accuracy of user portraits;

[0053] Step 2: After the user authorizes, the user obtains the data collection permission under the specified project in the user's personal data information database, performs data collection, and cleans the collected data;

[0054] Step 3: Select the required attributes of a certain application target. Users can select the required attributes according to the specific application scenario to make the generated user profile more targeted, thereby better meeting the needs of specific application targets;

[0055] Step 4: The judgment is selected. Based on the preset application target requirement attributes, a keyword crawling template is defined. The cleaned data is crawled with keywords using the crawling template, and then tagged and classified.

[0056] In the process of defining a keyword crawling template, the attributes of the keyword crawling template are: defining a list of keywords whose relevance meets the recognition threshold, setting context rules for keyword extraction, and searching and extracting through natural language processing technology;

[0057] Perform keyword crawling according to the defined keyword crawling template, extract keywords related to demand attributes, list synonyms, antonyms and related words of keywords, analyze semantic relevance, classify according to subject, attribute, importance, etc., and establish a keyword list of related needs, so that user portraits can reflect user characteristics in a more fine-grained manner;

[0058] Step 5: Obtain the classified label data, and based on the preset application target demand attributes, use the label mining algorithm, take all labels and application target demand indicators as input, obtain all labels that are strongly correlated with the application target demand indicators, and combine them as a user profile for a certain application target;

[0059] The operation process of the label mining algorithm is:

[0060] a. Extract features from the cleaned data, construct training samples, and divide them into training sets and test sets;

[0061] b. Build a label mining model through machine learning algorithms to analyze the correlation coefficient between input features and labels;

[0062] c. Predict new data based on the trained label mining model and generate labels;

[0063] Step 6: If it is not selected, customize the keyword crawling template, use the crawling template to crawl keywords for the cleaned data, and perform label classification;

[0064] In the process of customizing keyword crawling templates:

[0065] Actively set keywords for filtering and crawling data, determine the information crawling layout, specify crawling rules for each field, write code according to the crawling rules, crawl data and store it;

[0066] Test on different target web pages to see if the crawling effect meets expectations, and adjust the crawling rules based on the test results;

[0067] Step 7: Obtain the classified label data, apply the label mining algorithm, take all labels as input, analyze the preference features, output the preference features of each category label, and combine the various preference features as the current user profile;

[0068] In the process of outputting the preference features of each category, a clustering algorithm is used to group user tag data, calculate the similarity of each tag, obtain different representative tags, and analyze the representative tags’ corresponding preference feature topic categories. The application of the clustering algorithm makes the user portrait not only personalized, but also reflects the user’s preference features in different categories, helping the application to accurately locate user needs.

[0069] Compared with the existing technology, this method analyzes and processes user data layer by layer through multiple steps, making the generated user portrait more refined and personalized. It combines natural language processing and machine learning technology, can effectively process and analyze massive data, and improve the efficiency and accuracy of portrait generation;

[0070] Moreover, whether it is the keyword capture template automatically generated by the system or the keyword capture template customized by the user, it provides great flexibility to adapt to different application needs and scenarios, which not only improves the accuracy of the portrait, but also enhances the system's ability to respond to different user needs.

[0071] In other aspects, this embodiment also provides a user portrait generation device based on big data, such as Figure 2 As shown, including:

[0072] Storage module 1, used to obtain the read permission of user personal information, and to read and upload the collected and analyzed data;

[0073] Definition module 2, used for classifying the data obtained by storage module 1 and defining the attributes of the classified items;

[0074] Collection module 3, used to collect and clean data for designated projects based on project attributes in personal information;

[0075] The preset crawling module 4 is used to define a keyword crawling template according to preset requirements, crawl keywords for the cleaned data, and complete label classification;

[0076] A custom crawling module 5 is used to customize a keyword crawling template, and crawl keywords from the cleaned data through the custom template and perform label classification;

[0077] The preference analysis module 6 is used to obtain the data obtained by the custom crawling module 5, use the tag mining algorithm to analyze, output the preference characteristics of each category for all tags, and analyze the tendency of user behavior preferences in different fields;

[0078] A combination module 7 is used to apply a label mining algorithm based on the classified label data, analyze the association with the application target demand indicators, extract labels with strong association, and combine them;

[0079] By embedding the label data into a high-dimensional space, the label data is grouped using the K-means clustering algorithm to find the label group whose similarity with the target demand meets the threshold. The clustered label group is defined as having high internal similarity and strong relevance to the target demand.

[0080] The portrait generation module 8 is used to output a user portrait for a certain application target or behavioral preferences in different fields according to the trigger status of the preset capture module 4 and the custom capture module 5, store the multi-dimensional information through a graph database or a document database, and update the user portrait regularly or in real time.

[0081] The storage module 1 is interactively connected with the definition module 2 and the acquisition module 3 through a wireless network, the acquisition module 3 is interactively connected with the preset capture module 4 and the custom capture module 5 through a wireless network, the custom capture module 5 is interactively connected with the preference analysis module 6 through a wireless network, and the combination module 7 is interactively connected with the preset capture module 4, the preference analysis module 6 and the portrait generation module 8 through a wireless network.

[0082] In the specific implementation of this embodiment, the personal data information that the user is allowed to access is registered through the storage module 1, the definition module 2 performs active project classification, and then the collection module 3 performs special collection according to business requirements. If there is a specified target, the preset crawling module 4 is triggered to perform crawling behavior based on the preset requirements, the combination module 7 combines labels, and the portrait generation module 8 outputs the final user portrait based on the specified target. If there is no specified target, the custom crawling module 5 is triggered, the preference analysis module 6 analyzes the user's preferences, identifies and analyzes the preference characteristics, and then the combination module 7 combines labels, and the portrait generation module 8 outputs the user portrait based on the user's preferences as a reference for business recommendations.

[0083] This embodiment also proposes an electronic device, in which a memory stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor to implement a user portrait generation method based on big data.

[0084] In this embodiment, a computer-readable storage medium is also proposed, in which at least one instruction, at least one program, code set or instruction set is stored, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor to implement a user portrait generation method based on big data.

[0085] In summary, the present invention can deeply explore user behavior preferences through highly targeted data collection and analysis algorithms, so that user portraits can more accurately reflect the real needs and behaviors of individuals, comprehensively consider the different characteristics and behaviors of users, and provide a more comprehensive information basis for business decision-making, thereby supporting more effective personalized services and precision marketing;

[0086] Users can freely choose whether to preset application target demand attributes, adapt to different application scenarios and user preferences, and allow highly personalized user portrait generation, so that the present invention can provide customized solutions for the specific needs of different user groups.

[0087] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A user portrait generation method based on big data, characterized in that: The following steps are involved: Step 1: Define several project attributes based on the user's personal data information database; Step 2: After the user authorizes, the user obtains the data collection permission under the specified project in the user's personal data information database, performs data collection, and cleans the collected data; Step 3: Select the required attributes of an application target; Step 4: The judgment is selected. Based on the preset application target requirement attributes, a keyword crawling template is defined. The cleaned data is crawled with keywords using the crawling template, and then tagged and classified. Step 5: Obtain the classified label data, and based on the preset application target demand attributes, use the label mining algorithm, take all labels and application target demand indicators as input, obtain all labels that are strongly correlated with the application target demand indicators, and combine them as a user profile for a certain application target; Step 6: If it is not selected, customize the keyword crawling template, use the crawling template to crawl keywords for the cleaned data, and perform label classification; Step 7: Get the classified label data, apply the label mining algorithm, take all labels as input, analyze the preference features, output the corresponding preference features of each category label, and combine the various preference features as the current user portrait.

2. The method for generating user portraits based on big data according to claim 1, characterized in that: In the process of defining the keyword crawling template in step 4, the attributes of the keyword crawling template are: defining a keyword list whose relevance meets the recognition threshold, setting context rules for keyword extraction, and searching and extracting through natural language processing technology; Perform keyword crawling according to the defined keyword crawling template, extract keywords related to demand attributes, list the synonyms, antonyms and related words of the keywords, analyze the semantic relevance, classify according to subject, attribute, importance, etc., and establish a keyword list of related requirements.

3. The method for generating user portraits based on big data according to claim 1, characterized in that: The operation process of the label mining algorithm in step 5 is: a. Extract features from the cleaned data, construct training samples, and divide them into training sets and test sets; b. Build a label mining model through machine learning algorithms to analyze the correlation coefficient between input features and labels; c. Predict new data based on the trained label mining model and generate labels.

4. The method for generating user portraits based on big data according to claim 1, characterized in that: In the process of customizing the keyword crawling template in step 6: Actively set keywords for filtering and crawling data, determine the information crawling layout, specify crawling rules for each field, write code according to the crawling rules, crawl data and store it; Test on different target web pages to see if the crawling effect meets expectations, and adjust the crawling rules based on the test results.

5. The method for generating user portraits based on big data according to claim 1, characterized in that: In the process of outputting the preference features of each category in step 7, a clustering algorithm is used to group the user tag data, calculate the similarity of each tag, obtain different representative tags, and analyze the preference feature topic categories corresponding to the representative tags.

6. A user portrait generation device based on big data, wherein the method is a device equipped with a user portrait generation method based on big data according to any one of claims 1 to 5, characterized in that: include: Storage module 1, used to obtain the read permission of user personal information, and to read and upload the collected and analyzed data; Definition module 2, used for classifying the data obtained by storage module 1 and defining the attributes of the classified items; Collection module 3, used to collect and clean data for designated projects based on project attributes in personal information; The preset crawling module 4 is used to define a keyword crawling template according to preset requirements, crawl keywords for the cleaned data, and complete label classification; A custom crawling module 5 is used to customize a keyword crawling template, and crawl keywords from the cleaned data through the custom template and perform label classification; The preference analysis module 6 is used to obtain the data obtained by the custom crawling module 5, use the tag mining algorithm to analyze, output the preference characteristics of each category for all tags, and analyze the tendency of user behavior preferences in different fields; A combination module 7 is used to apply a label mining algorithm based on the classified label data, analyze the association with the application target demand indicators, extract labels with strong association, and combine them; The portrait generation module 8 is used to output a user portrait for a certain application target or behavioral preferences in different fields according to the trigger status of the preset capture module 4 and the custom capture module 5, store the multi-dimensional information through a graph database or a document database, and update the user portrait regularly or in real time.

7. The user portrait generation device based on big data according to claim 6, characterized in that: The combination module 7 embeds the label data into a high-dimensional space, groups the label data using a K-means clustering algorithm, and finds label groups whose similarity to the target demand meets a threshold. The clustered label groups are defined as having high internal similarity and strong correlation with the target demand.

8. The user portrait generation device based on big data according to claim 6, characterized in that: The storage module 1 is interactively connected with the definition module 2 and the acquisition module 3 through a wireless network, the acquisition module 3 is interactively connected with the preset capture module 4 and the custom capture module 5 through a wireless network, the custom capture module 5 is interactively connected with the preference analysis module 6 through a wireless network, and the combination module 7 is interactively connected with the preset capture module 4, the preference analysis module 6 and the portrait generation module 8 through a wireless network.

9. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the method as described in any one of claims 1-5.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the method described in any one of claims 1 to 5.

Images