Method for calculating resident portrait based on resident data relationship aggregation portrait model

Abstract

The present invention discloses a method for calculating a resident portrait based on a resident data relationship aggregation portrait model. The method includes: Step 1, sorting out and collecting resident information; Step 2, aggregating resident information data; Step 3, data synchronization service; Step 4, establishing tags; Step 5, establishing a user portrait; Step 6, tag data search; Step 7, tag calculation task service; Step 8, tag storage service; Step 9, failure compensation mechanism; Step 10, recalculation logic. This method makes the data in grass-roots governance work more accurate, facilitates querying and maintenance by grass-roots personnel, improves work efficiency, and enhances the satisfaction of grass-roots work.

Description

Technical Field

[0001] The present invention relates to the technical field of relationship graph recognition, and specifically, to a method for calculating a resident portrait based on a resident data relationship aggregation portrait model. Background Art

[0002] The management of the population base and the information of residents that need to be focused on in grass-roots sub-districts, towns, communities, and administrative villages has always been a difficult point in grass-roots governance. In previous actual work, grass-roots staff usually used methods such as excel or paper ledgers to record and manage, which is not convenient for querying and has very low timeliness. Specifically, the traditional resident portrait tags have the following problems:

[0003] 1. Maintaining a separate excel or paper ledger, or even an excel for each tag record. Such as: low-income elderly, single elderly, residents under jurisdiction control, etc., which is time-consuming and laborious;

[0004] 2. Difficult to count and compare. Because the data is very scattered, it is impossible to immediately know which portrait tag a person belongs to, nor can you always understand the information of various types of personnel and the overall residents in the current jurisdiction;

[0005] 3. Low timeliness, affecting work efficiency. Because each resident's information needs to be manually maintained, and the personnel classification, etc. also need to be statistically aggregated and maintained. Without a unified system, it brings a great workload to grass-roots personnel and increases a large amount of labor costs. Summary of the Invention

[0006] The purpose of the present invention is to provide a method for calculating a resident portrait based on a resident data relationship aggregation portrait model, which makes the data in grass-roots governance work more accurate, convenient for grass-roots personnel to query and maintain, improves work efficiency, and enhances the satisfaction of grass-roots work.

[0007] To achieve the above purpose, the present invention provides a method for calculating a resident portrait based on a resident data relationship aggregation portrait model, the method comprising:

[0008] Step 1, sorting out and collecting resident information;

[0009] Step 2, aggregating resident information data, using the data warehouse StarRocks to achieve real-time synchronization of massive data, and synchronizing the data to the relational database MySQL in real time;

[0010] Step 3, data synchronization service, completing the synchronization of the source data in the relational database MySQL to the StarRocks data table, including table structure, full amount of data and incremental data; at the same time, synchronizing incremental data through Kafka;

[0011] Step 4: Establish tags. Use the SQL statements generated by the tag rule service to calculate the tag source data in the database according to the rule SQL, and generate current tag data, tag historical data, and tag trace data;

[0012] Step 5: Establish user portraits, that is, label relevant user information. Analyze the characteristics and behaviors of users by collecting data relationships in various dimensions such as streets, towns, communities, and administrative villages; then combine the portrait label logic that grass-roots personnel need to pay attention to, and decompose the portrait into individual tag definition logics; finally, converge the tags into each resident's information, and each resident's unique portrait is composed of multiple tags;

[0013] Step 6: Tag data search, including providing tag detail data and tag aggregation data;

[0014] Step 7: Tag calculation task service. Among them, the task service provides task scheduling, management, and auditing services for tag calculation tasks;

[0015] Step 8: Tag storage service, providing storage services for tag detail data, tag historical data, tag difference data, and tag aggregation data;

[0016] Step 9: Failure compensation mechanism. For data that fails to be obtained in real time, use logical code to maintain and check it regularly, and use the compensation method to maintain the latest data in the data warehouse;

[0017] Step 10: Recalculation logic. After the data is modified, the data warehouse will promptly trigger the recalculation and statistical logic of the corresponding tags.

[0018] Preferably, the data sources of the resident information sorted and collected in Step 1 include information from the human resources and social security department, the civil affairs department, the health and health department, the public security department, and the education department.

[0019] Preferably, Step 3 includes:

[0020] Step 3.1: After receiving the task sent by the client, the FE will execute the corresponding task according to the content in the Kafka import task and return a response to the client;

[0021] Step 3.2: Import within one Task, and finally judge whether it is successful based on the execution result of the task; each Task consists of multiple Tasks, each Task is managed by a task Manager, and the task Manager creates new Tasks or converts existing Tasks into other tasks according to the needs of the task;

[0022] In step 3.3, each task is assigned to the corresponding BE. On the BE, each task is considered an import task and is imported using the Stream Load import mechanism. Each task is assigned to one or more data structures. If there are multiple data structures, they can be treated as separate jobs, with one as the primary job and the others as secondary jobs.

[0023] Step 3.4: BE is an extension based on FE and starts reporting to FE after BE is imported;

[0024] Step 3.5: FE generates a new task based on the feedback results, and retries the failed task if any;

[0025] Step 3.6: In order to complete the continuous import of data, FE will continuously generate new tasks.

[0026] Preferably, step 4 includes:

[0027] Step 4.1: Get the data source library and label library. The data source library stores the data to be labeled, and the label library stores the label information.

[0028] Step 4.2: Assign the value of the tag type to the corresponding attribute in the data source library so that the value of the tag can be recognized and processed by the database;

[0029] Step 4.3: Determine a model corresponding to the tag type and at least one tag library associated with the model;

[0030] Step 4.4: Use the model and the tag library bound to it to construct tags for the data that needs to be tagged.

[0031] Preferably, after generating labels in step 4, the labels are divided into the following categories according to the validity of the data:

[0032] Static labels refer to objective facts that will not change;

[0033] Dynamic tags refer to time-sensitive tags that need to be updated regularly;

[0034] Model labeling, no data, customized rules, requires later model building to derive labels;

[0035] Predictive tags: predict user behavior based on existing data.

[0036] Preferably, step 5 includes:

[0037] Step 5.1: Let f(x,y) represent the number of times user x has marked a tag y, and TF(x,y) represent the weight ratio that the marking times occupy among all tags of the current user. The formula is as follows: Among them, f(x,y) is the number of times of tag y, and ∑f(x,y’) represents the total number of all tags;

[0038] Step 5.2: The rarity of tag y among all tags is represented by IDF(x,y), that is, the occurrence probability of this tag: Among them, ∑∑f(x1,y1) represents the sum of all tags of the user, and ∑f(x1,y) is the sum of all users who have marked tag y; at this time, if the occurrence probability of a certain tag is very low and it is simultaneously used to mark a user, it indicates that the relationship between this user and this tag is very close; then, the weight coefficient of this user under this tag is calculated according to IDF*TF.

[0039] Preferably, the search steps in Step 6 successively include designing a table, designing fields, and creating a table.

[0040] Preferably, in Step 8, the execution method of the data is adjusted, including:

[0041] Storing in partitions and executing jobs separately;

[0042] Developing an intermediate table based on the homologous data of tags;

[0043] And optimizing the performance of tag scripts and intermediate table execution.

[0044] According to the above technical solutions, the present invention includes the following steps: 1. Aggregating resident data information based on relational databases and non-relational databases to identify multi-element information such as resident names, ID numbers, contact information, and address information; 2. Through the definition of resident portrait tags, using big data computing to refine the relationships between residents and houses, jurisdictions, organizations, etc. according to different levels such as towns / streets, communities, and grid communities, making grass-roots governance more refined and providing more accurate information; 3. Providing a big data computing statistics dashboard for superiors and accurate portrait data and resident information data ledgers for subordinates, and allocating different portrait tag permissions according to organizational roles, making the data in grass-roots governance work more accurate and facilitating grass-roots personnel to query and maintain.

[0045] Other features and advantages of the present invention will be described in detail in the subsequent specific implementation part. Description of the Drawings

[0046] The drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the following specific implementation, they are used to explain the present invention, but do not constitute a limitation to the present invention. In the drawings:

[0047] Figure 1 is a flowchart of a method for calculating a resident portrait based on a resident data relationship aggregation portrait model provided by the present invention;

[0048] Figure 2 is a flowchart of collecting, aggregating, tagging, and maintaining label data in the method for calculating a resident portrait based on a resident data relationship aggregation portrait model provided by the present invention. Specific Embodiments

[0049] The following will describe in detail the specific embodiments of the present invention with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for explaining and illustrating the present invention, and are not used to limit the present invention.

[0050] See Figure 1 , the present invention provides a method for calculating a resident portrait based on a resident data relationship aggregation portrait model, and the method includes:

[0051] Step 1, sort out and collect resident information;

[0052] Step 2, aggregate resident information data, use the data warehouse StarRocks to achieve real-time synchronization of massive data, and synchronize the data to the relational database MySQL in real time;

[0053] Step 3, data synchronization service, complete the synchronization of source data in the relational database MySQL to the StarRocks data table, including table structure, full amount of data, and incremental data; at the same time, synchronize incremental data through Kafka;

[0054] Step 4, establish labels, calculate the label source data in the database according to the rule SQL through the SQL statement generated by the label rule service, and generate current label data, label historical data, and label track data;

[0055] Step 5, establish a user portrait, that is, tag relevant user information, analyze the characteristics and behaviors of users by collecting data relationships in various dimensions such as streets, towns, communities, and administrative villages; then combine the portrait label logic that grass-roots personnel need to pay attention to, and decompose the portrait composition into individual label definition logics; finally, converge the labels into each resident information, and each resident's unique portrait is composed of multiple labels;

[0056] Step 6, label data search, including providing label detail data and label aggregation data;

[0057] Step 7, label calculation task service, wherein the task service provides task scheduling, management, and auditing services for label calculation tasks;

[0058] Step 8, Label Storage Service, which provides storage services for label detail data, label history data, label difference data, and label aggregation data;

[0059] Step 9, Failure Compensation Mechanism. For data that fails to be obtained in real time, logical codes are used to perform regular maintenance checks, and the latest data is maintained in the data warehouse in a compensatory manner;

[0060] Step 10, Recalculation Logic. After the data is modified, the data warehouse will promptly trigger the recalculation and statistical logic of the corresponding labels, greatly improving the data calculation efficiency and data accuracy.

[0061] Among them, the data sources of the resident information sorted and collected in Step 1 include information from the human resources and social security department, civil affairs department, health and health department, public security department, and education department.

[0062] The above-sorted resident information includes:

[0063] 1. Resident basic information (resident_base): mainly includes personal basic information such as name, ID card, mobile phone number, gender, age, and home address;

[0064] 2. Resident privacy information (resident_privacy): mainly includes photos, call records, text message records, address book information, chat records of IM application software, personal videos, etc.;

[0065] 3. Resident account information (resident_account): mainly includes online banking accounts, social accounts, etc.;

[0066] 4. Resident social relationship information (resident_social_relations): mainly includes family member information, friend relationships, work unit information, etc.;

[0067] 5. Resident network behavior information (resident_network_behavior): mainly refers to online behavior records, such as personal information like online time and online location.

[0068] In Step 2, the main problem is how to prevent the avalanche effect from occurring during the batch update of data in the metadata side, which may lead to data congestion. Instead, StarRocks is developed based on Doris. In the OLAP market, StarRocks is widely praised for its good performance in multi-table join queries and is a very useful data warehouse that can effectively solve the avalanche effect problem.

[0069] Also, how to standardly and efficiently build a data synchronization link so that implementation engineers can focus on the business itself without getting stuck in various technical details. Through technical demonstration, StarRocks, a data warehouse, is used to synchronize the relational database MySQL in real time. MySQL is used to maintain resident information, ensuring data accuracy and consistency.

[0070] Synchronizing various data sources from different business systems to the data warehouse is the basis for marking. Integrating data from different sources, in various forms and different dimensions together to obtain the latest and most comprehensive resident information.

[0071] Step 3 includes:

[0072] Step 3.1: After receiving the task sent by the client, the FE will execute the corresponding task according to the content in the Kafka import task and return a response to the client;

[0073] 1) Integrate Kafka into the MySQL server;

[0074] 2) Use Kafka as a data cache to improve MySQL performance;

[0075] 3) Reduce the system complexity by simplifying and reducing management work.

[0076] Step 3.2: Import within a Task, and finally judge whether it is successful based on the execution result of the task; each Task consists of multiple Tasks, each Task is managed by a task Manager, and the task Manager creates a new Task or converts an existing Task into another task according to the needs of the task;

[0077] Step 3.3: Each Task is assigned to the corresponding BE for running; on the BE, a Task is regarded as an import task and imported through the Stream Load import mechanism; each Task is assigned to one or more data structures; if there are multiple, they can be regarded as separate Jobs, and one of them is regarded as the main Job and the other as the secondary Job;

[0078] Step 3.4: The BE is an extension layer based on the FE and starts reporting to the FE after the import on the BE;

[0079] Step 3.5: The FE generates a new task based on the feedback result and retries the failed tasks if there are any;

[0080] Step 3.6: In order to complete the continuous import of data, the FE will continuously generate new tasks.

[0081] Through the above steps, the consistency and accuracy of the data are ensured. The data warehouse will actively synchronize the modified data once when the relational database changes, so as to ensure that the data warehouse can obtain the latest resident information.

[0082] Step 4 specifically includes:

[0083] Step 4.1: Obtain the data source library and the tag library. The data source library stores the data to be tagged, and the tag library stores the tag information.

[0084] Step 4.2: Assign the value of the tag type to the corresponding attribute in the data source library, so that the value of the tag can be recognized and processed by the database.

[0085] Step 4.3: Determine the model corresponding to the tag type, and determine at least one tag library bound to the model.

[0086] Step 4.4: Use the model and the bound tag library to construct tags for the data to be tagged.

[0087] At the same time, after generating tags in Step 4, the tags are classified according to the timeliness of the data into:

[0088] Static tags, which refer to objective facts that will not change.

[0089] Dynamic tags, which refer to tags with timeliness and need to be updated regularly.

[0090] Model tags, without data, with custom rules, and need to establish a model later to obtain tags.

[0091] Prediction tags, which predict the behavior of users based on existing data.

[0092] Tagging the user-related information in Step 5 includes:

[0093] Suppose there are the following tags and users in the community, and their relationship can reflect the relationship between tags to some extent, through:

[0094] Step 5.1: Use f(x,y) to represent the number of times user x tags a tag y, and TF(x,y) to represent the weight ratio that the tagging times occupy among all tags of the current user. The formula is: Among them, f(x,y) is the number of times of tag y, and ∑f(x,y’) represents the total number of tags;

[0095] The rarity of tag y among all tags is represented by IDF(x,y), that is, the occurrence probability of this tag: Among them, ∑∑f(x1,y1) represents the sum of all tags of a user, and ∑f(x1,y) is the sum of all users with the y tag; for example, if a user has 5 A tags, 2 B tags, and 1 C tag, then the TF of the A tag on this user is equal to 5 / (5 + 2 + 1). At this time, if the probability of a certain tag appearing is very low and it is used to mark users at the same time, it indicates that the relationship between this user and this tag is very close; then, the weight coefficient of this user under this tag is calculated according to IDF*TF. However, the result obtained here only focuses on the relationship between the user and the tag and does not combine the actual business, so it is necessary to combine various situations such as the specific business scenario of the tag.

[0096] In one embodiment:

[0097] User Wang has 3 X tags, 1 Y tag, and 2 Z tags on him;

[0098] User Zhang has 5 X tags and 2 Y tags on him;

[0099] User Zhu has 3 Z tags and 2 W tags on him.

[0100] It can be seen that both Wang and Zhang have the X and Y tags, which indicates that there is likely to be some correlation between X and Y. The more users and tags there are, the stronger the correlation between the tags.

[0101] At the same time, if a user has both the X tag and the Y tag, it is initially recorded as 1. As the amount of data increases, the more users there are between the two tags, the stronger the correlation between the two tags. Then, it can be considered that users with the X tag are very likely to also have the Y tag.

[0102] In addition, the search steps in step 6 include designing a table, designing fields, and creating a table in sequence. Specifically:

[0103] 1. Design of the table

[0104] Tag search consists of general tags and combined tags, and it is necessary to count the frequency of user search tags for confirmation. The general function is the up and down linkage between tags and selected tags. First of all, this tag is a tree structure and there is a type of parent-child cascade relationship, so that the tag data structure becomes clear and the database data is created.

[0105] 2. Design of the fields

[0106] TYPE: Based on the system requirements and considering the plug-and-play principle of tags, a field TYPE is provided to add an extension to the type of tags.

[0107] ID, FATHER_ID: Parent ID: FATHER_ID.

[0108] NAME, FATHER_NAME: Front-end display.

[0109] TITLE: Extended field.

[0110] TASK_ID: Business ID, used to distinguish different modules.

[0111] ORDER_ID: Sorting.

[0112] STATUS: Whether it is enabled.

[0113] QUICK_ID: Quick label.

[0114] IDSEARCH_FIELD: Search field.

[0115] HIGHLIGHT_COLOR: Customized highlighted field.

[0116] 3. Table creation statement

[0117]

[0118]

[0119] Through the above steps, search for a certain user and display the label data of this user.

[0120] In step 7, the following label calculations are provided:

[0121] Users belong to organizations, which are divided into streets, communities, and grids. Streets divide communities, and communities divide grids. Users have query permissions for data under their affiliated organizations. Organizations have spatial attributes, and the buildings within the scope of streets, communities, and grids are clear and hardly change, except when the organization is re-planned.

[0122] There are two association relationships between residents and organizations (streets, communities, grids), namely the housing relationship (buildings belong to the organization and are within the spatial scope under the jurisdiction of the organization) and the organizational relationship.

[0123] Housing relationship: There are residential, property right, and household registration relationships between residents and housing units, and the housing units belong to the grids planned by communities under the streets.

[0124] Organizational relationship: The Party organization belongs to the community, and residents belong to the Party organization.

[0125] If the label storage table runs continuously, it will consume a large amount of time. Therefore, the execution method of the data has been adjusted, including:

[0126] Partitioned storage and execute jobs separately;

[0127] Develop an intermediate table based on tag-based homologous data;

[0128] And optimize the performance of tag scripts and intermediate table execution.

[0129] In summary, this method is oriented to the multi-dimensional fine-grained user information of grass-roots streets, towns, communities, and administrative villages. It uses big data computing and utilizes the data warehouse StarRocks to achieve real-time analysis of massive data. Then, combined with the portrait tag logic that grass-roots personnel need to pay attention to, the decomposition of the portrait composition into individual tag definition logic methods is studied.

[0130] At the same time, based on a large amount of resident information in the relational database, combined with the personnel information recognition method, various types of information are aggregated using big data technology according to the resident portrait drawing logic required by the grass-roots level, and a resident portrait is drawn for each resident in the jurisdiction. Based on the relationship between the resident basic information and the housing information, various types of information of residents in different levels of jurisdictions are maintained and counted in real time, making the data in grass-roots governance work more accurate and facilitating grass-roots personnel to query and maintain.

[0131] The preferred embodiments of the present invention have been described in detail above in conjunction with the accompanying drawings. However, the present invention is not limited to the specific details in the above embodiments. Within the scope of the technical concept of the present invention, various simple modifications can be made to the technical solutions of the present invention, and these simple modifications all fall within the protection scope of the present invention.

[0132] In addition, it should be noted that, in the case of no contradiction, the various specific technical features described in the above specific embodiments can be combined in any suitable way. To avoid unnecessary repetition, the present invention will not separately describe various possible combination methods.

[0133] Furthermore, any combination can be made between various different embodiments of the present invention as long as it does not violate the idea of the present invention, and it should also be regarded as the content disclosed by the present invention.

Claims

1. A method for calculating a resident portrait based on a resident data relationship aggregation portrait model, characterized in that The method includes: Step 1: Sort out and collect resident information; Step 2: Aggregate resident information data, and use the data warehouse StarRocks to achieve real-time synchronization of massive data, and synchronize the data to the relational database MySQL in real time; Step 3: Data synchronization service, complete the synchronization of source data in the relational database MySQL to the StarRocks data table, including table structure, full amount of data and incremental data; at the same time, synchronize incremental data through Kafka; Step 4: Establish tags, and calculate the tag source data in the database according to the rule SQL through the SQL statement generated by the tag rule service to generate current tag data, tag historical data and tag track data; Step 5: Establish a user portrait, that is, label relevant information of the user, analyze the characteristics and behaviors of the user by collecting the data relationships of various dimensions of streets, towns, communities, and administrative villages; then combine the portrait tag logic that grass-roots personnel need to pay attention to, and decompose the portrait into individual tag definition logics; finally, converge the tags into each resident information, and form a unique portrait for each resident composed of multiple tags; Step 6: Tag data search, including providing tag detail data and tag aggregation data; Step 7: Tag calculation task service, where the task service provides task scheduling, management, and auditing services for tag calculation tasks; Step 8: Tag storage service, providing storage services for tag detail data, tag historical data, tag difference data, and tag aggregation data; Step 9: Failure compensation mechanism, for the data that fails to be obtained in real time, use logical code to maintain and check regularly, and use the compensation method to maintain the latest data in the data warehouse; Step 10: Recalculation logic, after the data is modified, the data warehouse will trigger the recalculation and statistical logic of the corresponding tags in a timely manner; among them, Step 5 includes: Step 5.

1. Let f(x, y) denote the number of times that user x has marked a tag y, and TF(x, y) denote the weight ratio that the marking times occupy among all tags of the current user. The formula is: , where f ( x,y ) is the number of times of tag y, and ∑ f ( x , y ’ ) represents the total number of tags; Step 5.

2. The rarity of tag y among all tags is represented by IDF(x, y), which is the occurrence probability of this tag: , where, ∑∑ f ( x 1, y 1) represents the sum of all tags of the user, and ∑ f ( x 1, y ) is the sum of all users tagged with y; at this time, if the occurrence probability of a certain tag is very low and it is used to label users at the same time, it indicates that the relationship between this user and this tag is very close; then, the weight coefficient of this user under this tag is calculated according to IDF * TF.

2. The method for calculating a resident portrait based on a resident data relationship aggregation portrait model according to claim 1, wherein The data sources of the resident information sorted out and collected in Step 1 include information from the human resources and social security department, civil affairs department, health and health department, information public security department, and education department.

3. The method for calculating a resident portrait based on a resident data relationship aggregation portrait model according to claim 1, wherein Step 3 includes: Step 3.1: After receiving the task sent by the client, FE will execute the corresponding task according to the content in the Kafka import task and return a response to the client; Step 3.2: Import within a Task, and finally judge whether it is successful based on the execution result of the task; each Task is composed of multiple Tasks, and each Task is managed by a task Manager. The task Manager creates new Tasks or converts existing Tasks into other tasks according to the needs of the task; Step 3.3: Each Task is assigned to the corresponding BE for running; on the BE, a Task is regarded as an import task and imported through the Stream Load import mechanism; each Task is assigned to one or more data structures; if there are multiple, it is regarded as a separate Job, and one of them is used as the main Job and the other as the auxiliary Job; Step 3.4: BE is an extension layer based on FE, and starts reporting to FE after the BE import; Step 3.5: FE generates a new task based on the feedback result and retries the failed tasks if any. Step 3.6: To complete the continuous import of data, FE continuously generates new tasks.

4. The method for calculating a resident portrait based on a resident data relationship aggregation portrait model according to claim 1, characterized in that, Step 4 includes: Step 4.1: Obtain the data source library and the tag library. The data source library stores the data to be tagged, and the tag library stores the tag information. Step 4.2: Assign the value of the tag type to the corresponding attribute in the data source library so that the value of the tag can be recognized and processed by the database. Step 4.3: Determine the model corresponding to the tag type and determine at least one tag library bound to the model. Step 4.4: Use the model and the bound tag library to construct tags for the data to be tagged.

5. The method for calculating a resident portrait based on the resident data relationship aggregation portrait model according to claim 4, wherein After generating tags in Step 4, the tags are classified according to the timeliness of the data into: Static tags, which refer to objective facts that do not change. Dynamic tags, which refer to tags with timeliness and need to be updated regularly. Model tags, with no data, self-defined rules, and require building a model later to obtain tags. Prediction tags, which predict user behavior based on existing data.

6. The method for calculating a resident portrait based on a resident data relationship aggregation portrait model according to claim 1, characterized in that The search steps in Step 6 sequentially include designing tables, designing fields, and creating tables.

7. The method for calculating a resident portrait based on a resident data relationship aggregation portrait model according to claim 1, wherein In Step 8, the execution method of the data is adjusted, including: Partitioned storage and each executes jobs separately. Develop intermediate tables based on the homologous data of tags. And optimize the performance of tag scripts and intermediate table execution.

Images