Community information processing method and system

Abstract

This invention proposes a community information processing method and system, including: integrating community resident information into a graph structure based on family units; storing the root node of the graph structure as an index and mapping the index address to a cache server; and, when retrieving community resident information, accessing the index location in the cache server using the index keyword and retrieving the required information based on that location. In community applications, this invention can better leverage family relationships among residents to optimize the storage and retrieval of resident information. Furthermore, the consistent hashing algorithm employed by GH-DCA offers excellent scalability and fault tolerance.

Description

Technical Field

[0001] The present invention belongs to the technical field of big data processing, and in particular relates to a community information processing method and system. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Data processing technology is a key branch of big data technology. Its purpose is to establish a corresponding data model for real-world objects. Based on this model, relevant information about these objects is converted and stored in a database for management. Furthermore, the data reading process also uses this model as a channel to convert the read object information into a language that can be understood in the real world. The performance of data processing technology depends on the efficiency of data storage and transmission.

[0004] Early data processing technologies were primarily based on relational databases. These databases, using strictly defined data models, converted real-world objects and their corresponding features into data and then manipulated it. If the database paradigm is properly implemented, it can accelerate data management with minimal redundancy. However, real-world object relationships are not static. Furthermore, the volume of data used for business integration has been growing at terabyte levels in recent years. Previous databases with limited capacity and data relationships no longer meet this demand, leading to the emergence of non-relational database management technologies. The HBase database is a typical example of a non-relational database. It manages data in a column-based format, adapting to complex and changing data structures. Compared to relational databases, HBase offers higher efficiency and real-time performance in specific application scenarios such as random queries, multi-table joins, and historical data analysis.

[0005] However, although the HBase database itself can expand the relationship between data in the form of column clusters, it cannot effectively use these explicit relationships to further optimize the storage structure and retrieval process.

[0006] In particular, in the community service scenario involved in the present invention, the retrieval of community resident information is usually performed on a person-by-person basis. Once retrieval based on family information is involved, the query efficiency of the community database will drop sharply.

[0007] Generally speaking, the retrieval process for family information in resident information is as follows: First, extract the keywords related to family information from the resident information, then use the keywords as a basis to search for resident information again, and continue to loop the above steps until the family set of the current resident is no longer expanded. After the above steps, the information of all family members of the current resident is retrieved. However, the above retrieval process will cause the time complexity to increase sharply due to the increase in the search depth, which is not conducive to real-time business docking. In addition, since the family relationship information in the resident information is relatively symmetrical (for example, community residents who are father and son), a lot of redundant family relationship information is repeatedly stored in the database, resulting in a waste of storage resources and is not conducive to the extension and expansion of later data.

[0008] Generally, during the retrieval process of community resident information, the index corresponding to the family relationship graph is first written into the cache. Then, by reading the index from the cache, the family relationship graph corresponding to each index is gradually retrieved to find community residents and their family information that meets the query criteria. During the process of writing the index into the cache, its address assignment is often random, which makes the index call process itself uncontrollable during the retrieval process, thereby reducing retrieval efficiency. Therefore, a hash algorithm is often used to establish a mapping relationship between the index and the cache server. Using the index keyword, the index's location in the cache can be quickly accessed, thereby accelerating the retrieval process of resident and family information. However, the hash algorithm has the disadvantage of not being well adapted to changes in cache server deployment. If a community service application needs to replace or add a cache server due to storage or performance requirements, the relative address of the index will change, and the large amount of cached data stored therein will need to be remapped using the hash algorithm, which will significantly increase the server's computational pressure. Summary of the Invention

[0009] In order to overcome the above-mentioned deficiencies of the prior art, the present invention provides a community information processing method based on a graph structure and a Hash data compression algorithm to optimize the storage process and retrieval efficiency of community resident information.

[0010] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0011] In a first aspect, a community information processing method is disclosed, comprising:

[0012] Integrate community resident information into a graph structure based on family units;

[0013] Store the root node of the graph structure as an index and map the index address to the cache server;

[0014] When retrieving community resident information, the index keyword is used to access the location of the index in the cache server and obtain the required information based on the location.

[0015] As an implementation method, community resident information is integrated into a graph structure based on households. The specific process is as follows:

[0016] Convert the current resident information and family relationship information into graph nodes respectively, and connect all family nodes with the current resident node, while marking the current resident node;

[0017] Repeat the above steps for all unmarked or unvisited family nodes until the nodes contained in the family relationship graph generated with the current resident information as the center are no longer expanded.

[0018] As a preferred implementation method, when converting the current resident information and family relationship information into graph nodes respectively, the data in the original community resident table in the database is read, and the community resident information is traversed starting from the first row of data to determine whether the current resident information has been accessed. If the current resident information has not been accessed, the current resident information and family relationship information are converted into graph nodes respectively.

[0019] As an implementation method, for each family relationship graph in the graph structure, the oldest resident node is used as the index corresponding to the family relationship graph, and the index is stored in an HBase table.

[0020] As an implementation method, the index address is mapped to the cache server, specifically:

[0021] The consistent hashing algorithm is used to organize the entire hash value space into a ring area to continuously store the family relationship graph index, and the server address is also arranged in the ring area;

[0022] When the index to be retrieved is located in the ring area through the Hash algorithm, the first server it encounters in the clockwise or counterclockwise direction is the server where the current index should be located. Therefore, the information of all community residents in the current index and the corresponding family relationship diagram is accessed in this server.

[0023] As an implementation method, if the cache server changes, it is only necessary to relocate the index stored in the changed server.

[0024] As an implementation method, when querying, a keyword to be queried is received and parsed, resident information and family information are searched through a graph index in a cache, and information that meets the keyword is fed back to the query client.

[0025] In a second aspect, a community information processing system is disclosed, comprising:

[0026] The graph structure arrangement module is configured to: integrate community resident information into a graph structure based on family units;

[0027] The cache module is configured to: store the root node of the graph structure as an index and map the index address to the cache server;

[0028] The retrieval module is configured to: when retrieving community resident information, access the location of the index in the cache server through the index keyword and obtain the required information based on the location.

[0029] The present invention utilizes the family relationships implicit in resident information and re-integrates community resident information into a graph structure based on the family unit. The graph structure can intuitively reflect the relationship between family members. At the same time, the root node of the graph structure will be stored in the database as an index for the family. In order to optimize the reading and writing operations of the index and enable the community database to better provide business support related to resident information, a consistent hash algorithm is used to map the index address to the cache server. When too much resident information is stored and the cache server needs to be adjusted, the consistent hash algorithm allows the mapping of some indexes to change without affecting the global index storage and retrieval performance, and is suitable for community-related application scenarios.

[0030] One or more of the above technical solutions have the following beneficial effects:

[0031] The present invention proposes a data compression algorithm based on graph and hash (hereinafter referred to as GH-DCA) to optimize the storage process and retrieval efficiency of community resident information.

[0032] GH-DCA reorganizes semi-structured resident information and integrates resident information by household in the form of a graph structure. At the same time, GH-DCA stores the root node of the graph structure as an index in the database, and uses a consistent hash algorithm to map the index to the cache server for query and call.

[0033] Compared with other data compression and storage algorithms, GH-DCA can better utilize family relationships between residents to optimize the storage and retrieval process of resident information in community application scenarios. In addition, the consistent hash algorithm adopted by GH-DCA has excellent scalability and fault tolerance. When too much resident information is stored and the cache server needs to be adjusted, the consistent hash algorithm allows the mapping of some indexes to change without affecting the global index storage and retrieval performance, making it suitable for community-related application scenarios.

[0034] Advantages of additional aspects of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0036] Figure 1 The overall flow chart of the method for compressing and storing community family information proposed in the embodiment of the present invention

[0037] Figure 2 The process of generating the family relationship diagram of community residents in the embodiment of the present invention

[0038] Figure 3 This is a schematic diagram of the consistent hash algorithm used in the present invention. DETAILED DESCRIPTION

[0039] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0040] It should be noted that the terms used herein are for describing particular embodiments only and are not intended to limit the exemplary embodiments according to the present invention.

[0041] In the absence of conflict, the embodiments of the present invention and the features thereof may be combined with each other.

[0042] Example 1

[0043] This embodiment discloses a method for optimizing the storage and retrieval efficiency of community resident information in an HBase-based community database, enabling community service applications to access relevant information about community residents in real time and achieving business integration between community service applications and the community database. Specifically, a data compression algorithm based on graph and hash (GH-DCA) is proposed to optimize the storage and retrieval efficiency of community resident information. GH-DCA first analyzes the existing community resident information in the HBase-based community database based on family relationships. Based on this analysis, community users are divided into family relationship graphs. Based on this analysis, an index corresponding to each family relationship graph is established to accelerate the retrieval process of community resident information and its family members. Furthermore, GH-DCA uses a consistent hashing algorithm to map the index corresponding to the family relationship graph to a cache server for easy query and retrieval. Compared to the original hashing algorithm, the consistent hashing algorithm maintains excellent fault tolerance and scalability even when the cache server deployment changes. If a cache server changes, only the index stored in the changed server needs to be relocated, while the data in other servers remains unaffected.

[0044] In summary, the GH-DCA method proposed in this paper can store community resident information in the form of a family relationship graph, while simultaneously eliminating redundant family relationship information to conserve database storage space. Furthermore, a consistent hashing algorithm is used to establish a cache address mapping for the family relationship graph index, thereby accelerating index access. Compared to other data compression and storage algorithms, GH-DCA can better utilize family relationships among residents in community applications to optimize the storage and retrieval of resident information, achieving improved query efficiency and stability.

[0045] The present invention optimizes the storage method and retrieval process of resident information to improve the business connection efficiency of community service applications.

[0046] The following are the technical steps involved in the present invention:

[0047] Step 1: Receive community resident information from the community database and analyze the family relationship information contained in the community resident information.

[0048] Step 2: Using the analysis results of family relationship information in step 1, use a graph structure to integrate resident information by family. During the integration process, the root node of the graph structure will be stored in the database as an index.

[0049] Step 3: Use the consistent hashing algorithm to optimize the index mapping process, speeding up the index access process while improving the fault tolerance and scalability of the cache server.

[0050] Step 4: Reconnect the optimized community database to the community service application to achieve business integration. At this time, the storage and retrieval efficiency of community resident information in the community database is improved compared to before.

[0051] Figure 1 This is an overall flow chart of the community family information compression and storage method proposed in the present invention. The HBase-based community database involved in the present invention stores semi-structured community resident information in the form of HBase tables, and the family relationship information in the resident information is stored in column clusters, and each element in the column cluster is related family information.

[0052] Regarding step 2, in order to make full use of the family relationship information of community residents, compress the storage space occupied by community resident information while accelerating the retrieval efficiency of community resident information, GH-DCA uses a graph structure to integrate resident information on a family basis, and optimizes the retrieval process of family information in turn.

[0053] The GH-DCA method proposed in this paper uses a graph structure to integrate resident information on a household basis, sequentially optimizing the family information retrieval process. First, the current resident information and family relationship information are converted into graph nodes. All family nodes are connected to the current resident node, and the current resident node is marked. Next, the above steps are repeated for all unmarked (i.e., unvisited) family nodes until the family relationship graph generated with the current resident information as the center contains no more nodes.

[0054] In summary, a family relationship graph for the current residents has been generated. This graph consolidates redundant family relationships into edges between two resident nodes, effectively compressing the family relationship information within the resident data, thereby saving storage space in the community database. Furthermore, for each family relationship graph, the oldest resident node is used as the corresponding index, and this index is stored in an HBase table. Through these operations, resident information is consolidated into multiple graph-structured data based on explicit family relationship information. At the same time, the family information traversal problem is transformed into a graph traversal problem, improving retrieval efficiency.

[0055] Specifically, the steps for establishing the family relationship diagram corresponding to each resident are as follows: Figure 2 Said:

[0056] 1. Read the data in the original community resident table in the community database and traverse the community resident information starting from the first row of data.

[0057] 2. Determine whether the current resident information has been accessed. If the current resident information has not been accessed, proceed to the next step; otherwise, jump to step 4.

[0058] 3. Convert the current resident information and family relationship information into graph nodes respectively, and connect all family nodes to the current resident node. At the same time, mark the current resident node to indicate that the neighbor family relationship of the current resident information has been fully explored.

[0059] 4. Repeat step 3 for each unmarked family member node until no new nodes are added to the graph. The resulting graph structure is the family relationship graph corresponding to the residents in step 2. One of the nodes in the family relationship graph is used as the index corresponding to the graph, and the index is stored in the HBase table. Generally, the oldest family member node or the community resident node that holds community property ownership is selected as the index of the family relationship graph.

[0060] 4. Go to step 2 and analyze the family relationship information of the next resident until the information of all community residents is traversed.

[0061] 5. Finally, the organized community resident information is stored in the HDFS file. The integrated community resident information is represented by multiple graph structures, each of which represents each family in the community.

[0062] In summary, a family relationship graph for all residents in the community has been generated. This graph consolidates redundant family relationships into edges between two resident nodes, effectively compressing the family relationship information within the resident data, thereby saving storage space in the community database. Furthermore, for each family relationship graph, the oldest resident node is used as the corresponding index, and this index is stored in an HBase table. Through this operation, resident information is consolidated into multiple graph-structured data based on explicit family relationship information. At the same time, the family information traversal problem is transformed into a graph traversal problem, improving retrieval efficiency.

[0063] In step 3, to accelerate access to the family relationship graph index itself, GH-DCA uses a hash algorithm to map the index address to the cache server. Cache servers can accelerate index access, but the storage requirements of the community database are not static. When the number of cache servers changes, the index address will also change, and the large amount of cached data stored in them will need to be remapped using the hash algorithm, which will significantly increase the server's computing pressure.

[0064] The GH-DCA method proposed in this invention uses the consistent hash algorithm to optimize the index mapping process. The consistent hash algorithm is applied to the mapping establishment process of the index address in the cache server. Figure 3 As shown, the consistent hashing algorithm organizes the entire hash value space into a ring-shaped area to continuously store the family relationship graph index, and the server addresses are also arranged within the ring-shaped area. When the index to be retrieved is located in the ring-shaped area through the hashing algorithm, the first server encountered in a clockwise or counterclockwise direction is the server that the current index should be located. Therefore, the information of all community residents in the current index and the corresponding family relationship graph is accessed on this server. If the cache server changes, only the index stored in the changed server needs to be relocated, and the data in other servers will not be affected.

[0065] In summary, the consistent hashing algorithm shows good fault tolerance and scalability in the cache server and is suitable for the storage of family relationship graph indexes.

[0066] Finally, the optimized community database is reconnected to community service applications. In an HBase-based community database, the proposed method reorganizes semi-structured resident information, integrating it by family using a graph structure. Simultaneously, this method stores the root node of the graph structure as an index in the database and uses a consistent hashing algorithm to map the family relationship graph index to a cache server for easy query and access. This optimization method significantly improves the storage and retrieval efficiency of community resident information.

[0067] The method proposed by the present invention ultimately optimizes the storage and retrieval efficiency of the database, and accelerates the business docking process between community service applications and the database. In the community database based on HBase, the method proposed by the present invention converts the original linear storage structure corresponding to the community resident information into a graph storage structure, while eliminating redundant family relationship information and converting the linear recursive retrieval problem into a graph retrieval problem, greatly improving the retrieval efficiency of community resident information and family information. In addition, the consistent hash algorithm can effectively map the family relationship graph index to the cache server, accelerate the calling process of the index itself, and can relocate the index at a low cost when the cache server deployment changes, showing good fault tolerance and scalability. Therefore, the method proposed by the present invention has both efficiency and stability.

[0068] Example 2

[0069] The purpose of this embodiment is to provide a computing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the program.

[0070] Example 3

[0071] The purpose of this embodiment is to provide a computer-readable storage medium.

[0072] A computer-readable storage medium stores a computer program, which, when executed by a processor, performs the steps of the above method.

[0073] Example 4

[0074] The purpose of this embodiment is to provide a community information processing system, including:

[0075] The graph structure arrangement module is configured to: integrate community resident information into a graph structure based on family units;

[0076] The cache module is configured to: store the root node of the graph structure as an index and map the index address to the cache server;

[0077] The retrieval module is configured to: when retrieving community resident information, access the location of the index in the cache server through the index keyword and obtain the required information based on the location.

[0078] Example No

[0079] The purpose of this embodiment is to provide a community information processing system, including: at least one client and multiple cache servers;

[0080] The client integrates community resident information into a graph structure based on family units;

[0081] The root node of the graph structure is stored as an index and the index address is mapped to the cache server.

[0082] When the client retrieves community resident information, it accesses the location of the index in the cache server through the index keyword, obtains the required information based on the location, and feeds it back to the client.

[0083] The purpose of this invention is to reintegrate the resident information in the community database according to certain rules to improve the retrieval efficiency of community resident information, enable community service applications to call relevant information of community residents in real time, and realize business docking between community service applications and community databases.

[0084] The resident information involved in the present invention has family relationships, and the family relationships are stored in a semi-structured HBase table. How to use explicit family relationships to integrate resident information on a family basis and optimize the storage and retrieval process of family indexes is the source of the technical solution improvement of the present invention.

[0085] In the community database, the family relationships in the resident information are stored in a column cluster, and each element in the column cluster is the relevant family information. Although the community database makes good use of the scalable characteristics of HBase column storage to store semi-structured resident information, the family relationships between community residents are not well reflected, and there are redundant items in the family information. Once the retrieval needs related to the residents' families are involved, the retrieval efficiency of the community database will decrease, which is not conducive to real-time business docking. Therefore, the present invention proposes a data compression algorithm based on graph structure and Hash (DataCompress Algorithm based on Graph and Hash, hereinafter referred to as GH-DCA) to optimize the storage process and retrieval efficiency of community resident information. GH-DCA reorganizes the semi-structured resident information and integrates the resident information in the form of a graph structure on a family basis. At the same time, GH-DCA stores the root node of the graph structure as an index in the database, and maps the index to the cache server using a consistent hash algorithm for query and call. Compared with other data compression and storage algorithms, GH-DCA can better utilize family relationships between residents to optimize the storage and retrieval process of resident information in community application scenarios. In addition, the consistent hash algorithm adopted by GH-DCA has excellent scalability and fault tolerance. When too much resident information is stored and the cache server needs to be adjusted, the consistent hash algorithm allows the mapping of some indexes to change without affecting the global index storage and retrieval performance, making it suitable for community-related application scenarios.

[0086] The steps involved in the apparatus of the above embodiment correspond to those of the method embodiment 1. For detailed implementation, please refer to the relevant description of embodiment 1. The term "computer-readable storage medium" should be understood to mean a single medium or multiple media containing one or more instruction sets; it should also be understood to include any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and causing the processor to perform any method of the present invention.

[0087] Those skilled in the art will appreciate that the modules or steps of the present invention described above can be implemented using a general-purpose computer device. Alternatively, they can be implemented using program code executable by a computing device, which can then be stored in a storage device and executed by the computing device. Alternatively, they can be fabricated into separate integrated circuit modules, or multiple modules or steps can be fabricated into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0088] Although the above describes the specific embodiments of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without any creative work are still within the scope of protection of the present invention.

Claims

1. A community information processing method, characterized in that: include: The community resident information is integrated into a graph structure based on the family as a unit. The specific process is as follows: the current resident information and family relationship information are converted into graph nodes respectively, and all family nodes are connected to the current resident node, and the current resident node is marked at the same time; when converting the current resident information and family relationship information into graph nodes respectively, the data in the original community resident table in the database is read, and the community resident information is traversed from the first row of data to determine whether the current resident information has been accessed. If the current resident information has not been accessed, the current resident information and family relationship information are converted into graph nodes respectively; Repeat the above steps for all unmarked or unvisited family nodes until the number of nodes in the family relationship graph generated with the current resident information as the center is no longer expanded; The root node of the graph structure is stored as an index and the index address is mapped to the cache server. Specifically, for each family relationship graph in the graph structure, the oldest resident node is used as the index corresponding to the family relationship graph, and the index is stored in the HBase table. When retrieving community resident information, the index keyword is used to access the location of the index in the cache server and obtain the required information based on the location; when querying, the query keyword is received and parsed, and the resident information and family information are searched through the graph index in the cache, and the information that meets the keyword is fed back to the query client; If the cache server changes, only the index stored in the changed server needs to be relocated.

2. A community information processing method according to claim 1, characterized in that: Map the index address to the cache server, specifically: The consistent hashing algorithm is used to organize the entire hash value space into a ring area to continuously store the family relationship graph index, and the server address is also arranged in the ring area; When the index to be retrieved is located in the ring area through the Hash algorithm, the first server it encounters in the clockwise or counterclockwise direction is the server where the current index should be located. Therefore, the information of all community residents in the current index and the corresponding family relationship diagram is accessed in this server.

3. A community information processing system, characterized in that: include: The graph structure arrangement module is configured to: integrate community resident information into a graph structure based on family units. The specific process is: converting the current resident information and family relationship information into graph nodes respectively, and connecting all family nodes with the current resident node, while marking the current resident node; wherein, when converting the current resident information and family relationship information into graph nodes respectively, the data in the original community resident table in the database is read, and the community resident information is traversed from the first row of data to determine whether the current resident information has been accessed. If the current resident information has not been accessed, the current resident information and family relationship information are converted into graph nodes respectively; Repeat the above steps for all unmarked or unvisited family nodes until the number of nodes in the family relationship graph generated with the current resident information as the center is no longer expanded; The cache module is configured to: store the root node of the graph structure as an index and map the index address to the cache server. Specifically, for each family relationship graph in the graph structure, the oldest resident node is used as the index corresponding to the family relationship graph, and the index is stored in the HBase table; The retrieval module is configured to: when retrieving community resident information, access the location of the index in the cache server through the index keyword and obtain the required information based on the location; when querying, receive the query keyword and parse the keyword, search for resident information and family information through the graph index in the cache, and feed back the information that meets the keyword to the query client; If the cache server changes, only the index stored in the changed server needs to be relocated.

4. A computing device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the method according to any one of claims 1 to 2 are implemented.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 2 are performed.

6. A community information processing system, characterized in that: include: At least one client and multiple cache servers; The client integrates community resident information into a graph structure based on family units. The specific process is as follows: converting the current resident information and family relationship information into graph nodes respectively, and connecting all family nodes with the current resident node, while marking the current resident node; wherein, when converting the current resident information and family relationship information into graph nodes respectively, reading the data in the original community resident table in the database, traversing the community resident information starting from the first row of data, and determining whether the current resident information has been accessed. If the current resident information has not been accessed, converting the current resident information and family relationship information into graph nodes respectively; Repeat the above steps for all unmarked or unvisited family nodes until the number of nodes in the family relationship graph generated with the current resident information as the center is no longer expanded; The root node of the graph structure is stored as an index and the index address is mapped to the cache server. Specifically, for each family relationship graph in the graph structure, the oldest resident node is used as the index corresponding to the family relationship graph, and the index is stored in the HBase table. When the client retrieves community resident information, it accesses the index location in the cache server through the index keyword, obtains the required information based on the location and feeds it back to the client; when querying, it receives the query keyword and parses the keyword, searches for resident information and family information through the graph index in the cache, and feeds back the information that meets the keyword to the query client; If the cache server changes, only the index stored in the changed server needs to be relocated.

Images