Extensible smart community knowledge graph construction method and system

Abstract

The invention provides an extensible smart community knowledge graph construction method and system, and relates to the technical field of artificial intelligence, and the method comprises the steps: carrying out the preprocessing of multi-modal perception data collected by a plurality of space regions in a smart community; according to a plurality of space areas preset in the smart community and physical adjacency relations thereof, constructing a space topological graph taking the space areas as nodes and the physical adjacency relations as edges; according to the method, the preprocessed multi-modal sensing data is processed to generate fusion features representing dynamic events in an area, the fusion features serve as attributes of corresponding nodes in a spatial topological graph, and a fusion graph forming a knowledge graph is constructed. According to the method, the relevance of the multi-modal data is enhanced through the spatial topological structure; and the interpretability and reasonability of the data are improved, so that the understanding and response capabilities of the intelligent space to complex interaction tasks are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a scalable method and system for constructing a knowledge graph for smart communities. Background Technology

[0002] With the popularization of IoT and AI technologies, smart spaces (such as smart homes and smart communities) collect multimodal data, including video, audio, and environmental parameters, through numerous sensors, aiming to achieve intelligent perception and response to user behavior and environmental conditions. However, current mainstream technologies face bottlenecks in understanding complex, context-dependent human activities. On the one hand, traditional multimodal fusion methods typically rely on strict spatiotemporal alignment, making it difficult to handle asynchronous and missing data in real-world scenarios. On the other hand, they often focus on low-level feature concatenation, lacking effective modeling of high-level dynamic semantics such as user behavior chains and task flows.

[0003] Some existing advanced solutions attempt to leverage large models for end-to-end scene understanding. For example, some solutions input visual and speech information into large language models (LLMs) for reasoning. However, such methods generally ignore the inherent physical topology and task context of the intelligent space, resulting in isolated and shallow understandings of events, making it difficult to support complex multi-step, cross-space tasks. Meanwhile, while knowledge graphs can represent entity relationships, current construction methods primarily rely on static knowledge and cannot effectively express dynamic causal chains centered on events and supported by multimodal data.

[0004] In summary, there is an urgent need in the current technological field for a new technical solution that can bridge the gap between unstructured multimodal perceptual data and structured knowledge representation. Specifically, there is a need for a knowledge representation method that can organically integrate spatial topology, temporal dynamics, and multimodal evidence, and combine it with the powerful semantic reasoning capabilities of large language models, thereby achieving an accurate, dynamic, and structured understanding of users' high-level intentions in intelligent spaces. Summary of the Invention

[0005] This invention provides a scalable method and system for constructing a knowledge graph for smart communities, which addresses the shortcomings of existing technologies such as static knowledge representation, lack of spatial context, insufficient multimodal fusion, and limited ability to understand complex tasks. It enables accurate modeling and semantic reasoning of dynamic events, behavioral chains, and user intentions in smart spaces.

[0006] This invention provides a scalable method for constructing a knowledge graph for smart communities, comprising: Preprocessing of multimodal sensing data collected from multiple spatial areas within the smart community; Based on the multiple spatial regions and their physical adjacency relationships within the smart community, a spatial topology graph is constructed with the spatial regions as nodes and the physical adjacency relationships as edges. The preprocessed multimodal sensing data is processed to generate fusion features that characterize dynamic events in the region, and the fusion features are used as attributes of corresponding nodes in the spatial topology graph to construct the fusion graph of the knowledge graph.

[0007] According to the scalable smart community knowledge graph construction method provided by the present invention, the multimodal perception data includes at least one of video data, audio data, and environmental sensor data; The preprocessing of multimodal sensing data collected from multiple spatial areas within the smart community includes: when the multimodal sensing data includes video data, uniformly sampling and normalizing the video stream; when the multimodal sensing data includes audio data, denoising and normalizing the audio data; and when the multimodal sensing data includes environmental sensor data, using a low-pass filter to filter, denoise, and normalize the environmental sensor data.

[0008] According to the scalable smart community knowledge graph construction method provided by the present invention, a spatial topology graph is constructed based on multiple preset spatial regions within the smart community and their physical adjacency relationships, with the spatial regions as nodes and the physical adjacency relationships as edges, specifically including: The spatial regions are divided according to the physical layout of the smart community and serve as nodes in the spatial topology map. Sensors deployed in each spatial region are assigned to the nodes based on their physical locations. Physical adjacency relationships between nodes are established based on the existence of physical connectivity between spatial regions, so that the nodes and the physical adjacency relationships together construct the spatial topology map.

[0009] According to the scalable smart community knowledge graph construction method provided by the present invention, the preprocessed multimodal sensing data is processed to generate fusion features representing dynamic events in the region, specifically including: The multimodal perception data is temporally encoded within a preset time window to generate temporal representations of each modality; the temporal representations of each modality are compressed to generate compressed features of each modality; the compressed features of each modality are fused across modalities to generate fused features; the cross-modal fusion is achieved by using a set of learnable query vectors and aggregating them with the compressed features of each modality through cross-attention interaction to generate the fused features.

[0010] According to the scalable smart community knowledge graph construction method provided by the present invention, after constructing the fusion graph that forms the knowledge graph, the method further includes: converting the information in the fusion graph into a natural language description; and inputting the natural language description into a pre-trained large language model for semantic reasoning to generate a structured recognition result or behavior prediction of the user's intent.

[0011] According to the scalable smart community knowledge graph construction method provided by the present invention, the information in the fused graph is converted into natural language description, specifically including: A graph neural network is applied to the fused graph, and the contextual semantics of the fused features of each node are enhanced through the adjacency propagation mechanism to obtain the enhanced fused features; The spatial topological relationship between the target node and its neighboring nodes, along with the enhanced fusion features, are combined and converted into the natural language description.

[0012] According to the scalable smart community knowledge graph construction method provided by the present invention, the method further includes: injecting the semantic reasoning results generated by the large language model as new knowledge feedback into the fusion graph to dynamically update the attributes of the corresponding nodes.

[0013] This invention also provides a scalable smart community knowledge graph construction system, comprising: The preprocessing module is used to preprocess the multimodal sensing data collected from multiple spatial areas within the smart community. The topology graph construction module is used to construct a spatial topology graph with the spatial regions as nodes and the physical adjacency relationships as edges, based on multiple preset spatial regions within the smart community and their physical adjacency relationships. The knowledge graph construction module is used to process the preprocessed multimodal perception data to generate fusion features that characterize dynamic events in the region, and to use the fusion features as attributes of corresponding nodes in the spatial topology graph to construct the fusion graph of the knowledge graph.

[0014] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the scalable smart community knowledge graph construction method described above.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the scalable smart community knowledge graph construction method as described above.

[0016] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the scalable smart community knowledge graph construction method as described above.

[0017] The scalable smart community knowledge graph construction method and system provided by this invention organizes multimodal perception data from multiple spatial areas within a smart community in a structured manner by constructing a spatial topology graph. This enables accurate representation of dynamic events, behavioral chains, and user intentions. The method utilizes spatial topology to enhance the correlation of multimodal data, improve the interpretability and reasonability of the data, and thus significantly improve the ability of the smart space to understand and respond to complex interactive tasks. It can more flexibly and effectively adapt to the diverse needs of residents, thereby enhancing the intelligence level and service efficiency of the smart community. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 This is one of the flowcharts illustrating the scalable smart community knowledge graph construction method provided by this invention.

[0020] Figure 2 This is a schematic diagram of the process for constructing a spatial topology map provided by the present invention.

[0021] Figure 3 This is a schematic diagram of the generation process of the fusion feature provided by the present invention.

[0022] Figure 4 This is a schematic diagram illustrating the generation of the fusion feature provided by the present invention.

[0023] Figure 5 This is a flowchart illustrating the semantic reasoning process using a large language model provided by the present invention.

[0024] Figure 6 This is the second flowchart illustrating the scalable smart community knowledge graph construction method provided by this invention.

[0025] Figure 7 This is a schematic diagram of the scalable smart community knowledge graph construction system provided by the present invention.

[0026] Figure 8 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0028] In smart space scenarios (such as smart communities and smart homes), the collection and processing of multimodal sensing data is fundamental to realizing intelligent services. This data includes video, audio, and environmental parameters (such as temperature, humidity, and light intensity), which are collected through devices such as cameras, microphone arrays, and sensors. However, this data exhibits several characteristics during actual collection, such as asynchronous sampling (different modalities are collected at different frequencies), data incompleteness (some modal data may be missing due to equipment failure or environmental interference), and semantic bias (different modalities have different semantic expressions). These characteristics make traditional multimodal fusion methods difficult to handle effectively, especially when modeling and understanding complex interactive tasks.

[0029] Furthermore, user behaviors and events in smart spaces are often dynamic and context-dependent. For example, a user's activities between different spatial areas (such as the living room, kitchen, and bedroom) may involve multiple steps and tasks, and these behavioral chains and task flows need to be modeled and reasoned across time and space dimensions. However, existing knowledge graph construction methods mainly focus on static entity relationships and lack effective representations of dynamic events and behavioral chains. Meanwhile, while large language models possess powerful semantic understanding capabilities, their input is typically natural language, making it difficult to directly process graph structures or multimodal perception data, thus limiting their application in smart spaces.

[0030] In summary, existing technologies fail to fully consider the dynamic nature of data, spatial contextual relationships, and semantic structure of complex tasks when processing multimodal sensing data in intelligent spaces. This limits the ability of intelligent space systems to accurately perceive and respond to user behavior and environmental conditions.

[0031] To address the aforementioned issues, this invention proposes a scalable method and system for constructing a knowledge graph for smart communities. This method introduces spatial topology modeling, associates multimodal sensing data with spatial nodes, and combines the contextual semantic enhancement of graph neural networks with the semantic reasoning capabilities of large language models to achieve accurate modeling and reasoning of dynamic events, behavioral chains, and user intentions in smart spaces.

[0032] Before detailing the technical solutions of the embodiments of the present invention, the terms and concepts involved in the embodiments of the present invention will be explained illustratively.

[0033] Intelligent space: refers to a spatial environment that uses technologies such as the Internet of Things, sensors, and artificial intelligence to perceive, analyze, make decisions, and execute automatically in real time within the physical environment.

[0034] Multimodal sensing data refers to data types from multiple heterogeneous sensing channels, including video, audio, text, environmental parameters, etc., which are asynchronous in the spatiotemporal dimensions.

[0035] Spatial topology diagram: refers to the physical connection relationship or functional linkage relationship between entities such as devices, areas, and people in a smart space, which is usually modeled in a graph structure.

[0036] Knowledge graph: A structured knowledge representation method that uses triples (entity, relation, entity) to represent semantic relationships between entities.

[0037] Transformer: A deep learning model architecture based on self-attention mechanism.

[0038] Large Language Model (LLM): A pre-trained, large-scale language model used for natural language processing tasks.

[0039] The scalable smart community knowledge graph construction method and system of this invention are implemented through a collaborative hardware architecture, mainly including a sensor network, a data processing unit, a computing server, and user interaction devices. The sensor network is deployed in various areas of the smart community, encompassing various types such as cameras, microphone arrays, and environmental sensors, for real-time acquisition of multimodal sensing data such as video, audio, temperature, humidity, and light intensity, and transmits the data to the data processing unit via wired or wireless networks. The data processing unit is responsible for receiving, storing, and preprocessing this data, including operations such as normalization, noise reduction, and filtering to eliminate noise and outliers, and indexing and managing the data to support subsequent analysis.

[0040] As the core execution entity of the system, the computing server undertakes the tasks of knowledge graph construction and reasoning. It first constructs a spatial topology map based on the physical layout and sensor deployment information of the smart community. Then, it performs temporal encoding and cross-modal fusion on the preprocessed multimodal data to generate fused features, which are embedded into the spatial topology map to form a fused graph. Based on this, the computing server enhances the contextual semantics of the fused graph using a graph neural network and converts the information into natural language descriptions using a graph summarization module, which are then input into a large language model for semantic reasoning. The reasoning results are not only used to generate structured recognition of user intent or behavioral prediction, but also fed back into the fused graph to dynamically update its state, supporting multi-turn interactions and complex task reasoning.

[0041] User interaction devices are used to enable real-time interaction with users. These include input devices such as voice assistants, touchscreens, and mobile devices, as well as output devices such as displays and speakers. Users send commands or requests through input devices, and the system's response is displayed to the user through output devices. User interaction devices are connected to computing servers via networks to ensure the flow of information and interactive responses between users and the system.

[0042] In actual operation, the data collected by the sensor network is preprocessed by the data processing unit and then transmitted to the computing server. The computing server completes the knowledge graph construction and reasoning tasks, and feeds back the results to the user through the user interaction device. This hardware architecture not only supports efficient processing and reasoning of large-scale multimodal data, but also has good scalability and flexibility, and can adapt to different smart community scenarios and resource constraints, providing a solid hardware foundation for accurate modeling and reasoning of dynamic events, behavioral chains, and user intentions in smart spaces.

[0043] The following is combined Figures 1-7 This invention describes a scalable smart community knowledge graph construction method and system according to embodiments of the present invention.

[0044] Figure 1 This is a flowchart illustrating the scalable smart community knowledge graph construction method provided by the present invention, which includes: Step 101: Preprocess the multimodal sensing data collected from multiple spatial areas within the smart community.

[0045] In smart communities, various sensors are deployed in multiple spatial areas (such as living rooms, kitchens, bedrooms, public corridors, and activity rooms) to collect multimodal sensing data in real time. This data includes, but is not limited to, video data, audio data, and environmental sensor data. To ensure the accuracy and efficiency of subsequent processing, this multimodal sensing data first needs to be preprocessed.

[0046] The main purpose of preprocessing is to clean, standardize, and format the collected raw data to eliminate noise, unify the data format, and extract useful information. The specific steps of preprocessing are as follows: In a smart community, multimodal data collected by sensor networks is first transmitted to a data processing unit. Precise timestamps are appended to this data during collection for subsequent time-series analysis and synchronization. Upon receiving this data, the data processing unit performs preliminary formatting to ensure data integrity and consistency.

[0047] The collected multimodal data is normalized to convert it to a uniform numerical range for subsequent analysis and processing. Simultaneously, noise reduction is performed to remove potential environmental noise and outliers. This process improves data quality and reduces noise interference in subsequent analysis.

[0048] The preprocessed data is stored in a local or cloud database and indexed. This index allows for the rapid retrieval and extraction of data within a specific timeframe or spatial region, providing efficient data support for subsequent analysis tasks.

[0049] Through the aforementioned preprocessing steps, the multimodal sensing data collected in the smart community is cleaned, standardized, and formatted, providing high-quality and consistent input data for subsequent spatial topology map construction, fusion map generation, and semantic reasoning. The preprocessed data not only reduces noise interference but also unifies the data format, improving data usability and processing efficiency, laying a solid foundation for accurate modeling and reasoning of dynamic events, behavioral chains, and user intentions in the smart space.

[0050] Step 102: Based on the multiple spatial regions and their physical adjacency relationships preset within the smart community, construct a spatial topology graph with the spatial regions as nodes and the physical adjacency relationships as edges.

[0051] In smart communities, the construction of spatial topology maps is fundamental for achieving multimodal data fusion and dynamic event modeling. In this embodiment, a smart community refers to a community environment that achieves intelligent management through technologies such as the Internet of Things (IoT), sensor networks, and artificial intelligence (AI). It includes residential units, public facilities, and activity areas. Spatial areas refer to areas within the community with clearly defined functions or physical boundaries, such as the living room, kitchen, and bedroom within a residential unit, and public areas such as corridors, elevator lobbies, and activity rooms. These areas are the basic building blocks of a smart community, and they exist in a "physical adjacency relationship," meaning whether there is a direct physical connection or passageway between two spatial areas. For example, the living room and kitchen are connected by a door, or two residential units are connected by a corridor.

[0052] When constructing a spatial topology graph, nodes represent spatial areas within the smart community, with each node corresponding to a specific area, such as a living room or kitchen. Edges represent the physical adjacency between spatial areas. If there is a direct physical connection between two spatial areas, an edge is drawn between these two nodes. For example, in a residential unit, there is a door between the living room and the kitchen, which means that there is a physical adjacency between the living room and the kitchen. Therefore, in the topology graph, there will be an edge connecting the living room node and the kitchen node.

[0053] This step involves constructing a spatial topology map based on multiple pre-defined spatial areas within the smart community and their physical adjacencies. The spatial areas serve as nodes, and the physical adjacencies as edges. For example, in a residential unit, the living room is one node, the kitchen is another, and the door between the living room and kitchen is one edge, thus forming a simple spatial topology map. In this way, the spatial topology map clearly represents the layout and connections of various spatial areas within the smart community, providing a basic framework for multimodal data fusion and dynamic event modeling in spatial perception.

[0054] Step 103: Process the preprocessed multimodal sensing data to generate fusion features that characterize dynamic events in the region, and use the fusion features as attributes of corresponding nodes in the spatial topology graph to construct the fusion graph of the knowledge graph.

[0055] The core task of step 103 is to transform the preprocessed multimodal data into fusion features that can characterize dynamic events within the region, and to assign these fusion features to the corresponding nodes in the spatial topology graph, thereby constructing a fusion graph of the knowledge graph. This process aims to effectively integrate data from different modalities to form a structured knowledge representation that reflects dynamic events.

[0056] First, after preprocessing, multimodal sensing data includes various types of information such as video, audio, and environmental parameters. Although this data has undergone initial cleaning and standardization, further processing is still needed to extract features that characterize dynamic events. For example, video data can extract features such as the trajectory of people's movements and the movement of objects; audio data can extract features such as speech content and the type of background noise; and environmental parameter data can extract features such as temperature changes and changes in light intensity. These features reflect specific events occurring within the area, such as people entering and leaving, equipment usage, and environmental changes.

[0057] Next, these features from different modalities are fused using specific algorithms or models. The purpose of fusion is to integrate information from different perceptual channels to form a comprehensive feature representation, thereby more fully reflecting dynamic events within the area. For example, by combining the movement trajectory of a person in a video with the speech content in an audio file, it is possible to more accurately determine whether an event occurring within the area is a person conversing with others or engaging in some activity alone. This fused feature not only includes information from a single modality but also reflects the correlation and interaction between different modalities.

[0058] Finally, the generated fusion features are used as attributes of the corresponding nodes in the spatial topology graph. In the spatial topology graph, each node represents a spatial region, and the fusion features endow these nodes with semantic information about dynamic events. For example, the fusion features of the living room node may include information such as the type of human activity and changes in environmental parameters, reflecting specific events occurring in the living room. By combining fusion features with the spatial topology graph, the constructed knowledge graph's fusion map not only possesses a spatial structure but also reflects the occurrence and development of dynamic events within the region.

[0059] Through step 103, the knowledge graph of the smart community can more comprehensively and accurately represent the dynamic events of various spatial areas within the community in the form of a fused graph. This fused graph provides rich semantic information for multimodal data fusion for spatial perception and dynamic event modeling, laying a solid foundation for subsequent intelligent analysis and decision-making.

[0060] The scalable smart community knowledge graph construction method provided in this invention constructs a spatial topology graph to structurally organize multimodal perception data from multiple spatial areas within a smart community. This enables accurate representation of dynamic events, behavioral chains, and user intentions. The method utilizes spatial topology to enhance the correlation of multimodal data, improve the interpretability and reasonability of the data, and thus significantly improve the ability of the smart space to understand and respond to complex interactive tasks. This allows the system to adapt to the diverse needs of residents more flexibly and effectively, enhancing the intelligence level and service efficiency of the smart community.

[0061] Furthermore, in smart communities, the preprocessing of multimodal sensing data is a crucial step in realizing subsequent knowledge graph construction and dynamic event modeling. This data includes video data, audio data, and environmental sensor data, collected through a sensor network deployed in various spatial areas of the smart community. To ensure data quality and consistency, targeted preprocessing of this multimodal sensing data is necessary.

[0062] For video data, the preprocessing process includes uniform sampling and normalization of the video stream. Specifically, the video data is captured at a frame rate of 30 frames per second using a fixed-mount camera. The total number of frames after sampling is N, and the time interval between video frames is... The video frames, after being uniformly sampled, are divided into days to obtain a daily frame sequence. Where T is the total number of frames for the day. To reduce data volume and improve processing efficiency, the system samples the video stream uniformly, for example, extracting one frame per second. The sampled video frames are then resized to a uniform image size, such as 224×224 pixels, to suit the input requirements of subsequent deep learning models.

[0063] Subsequently, the video frames are normalized using the following formula to normalize the pixel values ​​of each frame:

[0064] in, and These are the mean and standard deviation of the pixel sequence of the video frames, respectively. This represents the raw video frame data at time t. This process standardizes the pixel values ​​of the video data to a uniform range, reducing differences between different video sources and providing high-quality input for subsequent feature extraction and analysis.

[0065] For audio data, the preprocessing includes noise reduction and normalization. The audio data is acquired by a microphone array at a sampling rate of 44.1 kHz. Since the audio data may contain ambient noise and echoes, the system first performs noise reduction using the Wav2Vec2 model to remove these interferences, thereby improving the quality of the audio signal.

[0066] The denoised audio signal is further normalized using the following formula to normalize the amplitude value of the audio signal:

[0067] in, and These are the mean and standard deviation of the audio signal, respectively. Let be the amplitude value of the audio signal at time point t. Normalized audio data is better suited for subsequent speech recognition and semantic analysis tasks, ensuring the comparability and consistency of audio data under different environments.

[0068] For environmental sensor data, the preprocessing includes filtering and normalization. The data collected by environmental sensors includes temperature (°C), humidity (%), and light intensity (lux). This data may be affected by environmental interference or sensor accuracy during acquisition, thus requiring filtering. The system uses a low-pass filter to filter and denoise the environmental sensor data to smooth short-term fluctuations and remove high-frequency noise.

[0069] The filtered environmental data is further normalized using the following formula to normalize the raw measurements for each mode:

[0070] in, and These are the mean and standard deviation of the modal data, respectively. These are the original measurements at time t. Normalized environmental data better reflect changes in environmental conditions, providing standardized input for subsequent environmental state analysis and behavioral correlation.

[0071] At the end of each 24-hour period, all video, audio, and environmental data for the day are uniformly stored and managed according to timestamps, providing daily-divided datasets for subsequent analysis tasks. Real-time data processing and real-time decision-making will be based on these daily datasets.

[0072] Through the aforementioned preprocessing steps, the multimodal sensing data collected in the smart community is cleaned, standardized, and formatted, providing high-quality and consistent input data for subsequent spatial topology map construction, fusion map generation, and semantic reasoning. The preprocessed data not only reduces noise interference but also unifies the data format, improving data usability and processing efficiency, laying a solid foundation for accurate modeling and reasoning of dynamic events, behavioral chains, and user intentions in the smart space.

[0073] Furthermore, in smart communities, constructing a spatial topology map is fundamental for achieving multimodal data fusion and dynamic event modeling. The spatial topology map uses spatial regions as nodes and the physical adjacencies between these regions as edges, intuitively reflecting the layout and connectivity of various areas within the community. (See also...) Figure 2 The specific implementation steps for constructing a spatial topology map include: 201. Divide the spatial areas according to the physical layout of the smart community, and use them as nodes in the spatial topology map.

[0074] First, based on the physical layout information of the smart community, the community is divided into multiple pre-defined spatial zones. These zones can be functional areas within residential units, such as living rooms, kitchens, bedrooms, and bathrooms, or public areas, such as corridors, elevator lobbies, activity rooms, and parking lots. Each spatial zone node... R i Defined by its three-dimensional boundary coordinates, it is expressed as: .

[0075] in, This represents the minimum coordinate point in the region. This represents the maximum coordinate point in the region.

[0076] For example, a residential unit can be divided into the following spatial areas: Living Room, Kitchen, Bedroom, Bathroom, and Corridor. These spatial areas serve as nodes in a spatial topology graph, with each spatial area node... R iIt represents a specific spatial area and is assigned a unique identifier. For example, a living room node can be represented as... R 1 The kitchen node can be represented as R 2 .

[0077] 202. Assign sensor locations to nodes based on their physical locations.

[0078] In smart communities, various sensors (such as cameras, microphone arrays, and environmental sensors) are deployed in different spatial areas. To associate sensors with these spatial areas, they need to be assigned to the corresponding spatial area nodes based on their physical location. The specific steps are as follows: Determine each sensor S j physical location coordinates (x j ,y j ,z j ).

[0079] Based on the sensor's location, it is mapped to the nearest spatial region node. R i Belonging function Defined as: .

[0080] In this way, each sensor is mapped to its corresponding spatial region node, forming a sensor set within each region. : .

[0081] For example, if a camera is installed on the ceiling of the living room and its location coordinates are within the three-dimensional boundary of the living room, then the camera is assigned to the living room node. R 1.

[0082] 203. Establish physical adjacency relationships between nodes based on whether there is physical connectivity between each spatial region, so as to construct the spatial topology graph by combining the nodes and the physical adjacency relationships.

[0083] Physical adjacency between spatial regions refers to the existence of a direct physical connection or path between two regions. For example, a living room and kitchen might be connected by a door, or two residential units might be connected by a corridor. These physical connectivity relationships are represented as edges in the topology graph, indicating the connectivity between nodes. The specific steps are as follows: Determine the physical connectivity between different spatial regions. If two spatial regions... R p and R q The boundary has a passable area, and the width of the overlapping area of ​​its boundaries is...w pq Greater than or equal to the minimum passage threshold If the two regions are considered to be physically adjacent, then they are considered to be physically adjacent. 。

[0084] Under the above conditions, in the spatial region R p and R q Establish an edge between them e pq .

[0085] For example, if the width of the door between the living room and the kitchen is greater than the passage threshold. Then at the living room node R 1 and kitchen node R Establish an edge between 2 e 12 .

[0086] By using the above steps, the divided spatial regions are treated as nodes, and physical adjacency relationships are used as edges, ultimately constructing a complete spatial topology graph. G =( V , E A), where: V={v1,v2,…,v n} represents a set of nodes in a spatial region; E represents the set of adjacency edges between nodes; A i Represents each node v i The set of attributes used to characterize the sensing capabilities of the region includes: sensor type distribution. Number of sensors | A i |, density perceived per unit volume Perception update frequency set F i The functional area label to which this area belongs.

[0087] For example, living room node R 1 The set of attributes may include: Sensor types: camera, microphone; Number of sensors: 2; Sensing density per unit volume: 0.5 (assuming a living room volume of 100 cubic meters and 2 sensors). Perception update frequency: Once per second; Functional area label: Residential area.

[0088] Through the above steps, the constructed spatial topology map can not only clearly represent the layout and connection relationships of various spatial areas within the smart community, but also provide an important structured foundation for multimodal data fusion and dynamic event modeling for spatial perception.

[0089] Furthermore, to achieve accurate modeling and understanding of dynamic events, the preprocessed multimodal sensing data needs further processing to generate fused features that characterize dynamic events within the region. This process includes three key steps: temporal encoding, compression processing, and cross-modal fusion, such as... Figure 3 As shown: 301. Perform time-series encoding on the multimodal sensing data within a preset time window to generate time-series representations of each modality.

[0090] Within a preset time window, the preprocessed multimodal sensing data is temporally encoded to generate temporal representations of each modality. This process aims to capture the dynamic changes of the data over time, providing a foundation for subsequent feature compression and fusion.

[0091] Temporal coding of video data: A sliding window temporal Transformer is used to process the video frame sequence. The sliding window mechanism can effectively capture the temporal relationships between video frames, while the Transformer architecture can utilize a self-attention mechanism to model inter-frame dependencies. In this way, a temporal representation r of the video modality is generated. t f .

[0092] Temporal coding of audio data: A bidirectional GRU (Gated Recurrent Unit) network is used to process the audio feature sequence. The bidirectional GRU can simultaneously consider the forward and backward dependencies of the audio signal, thus better capturing the temporal characteristics of the audio signal. In this way, a temporal representation r of the audio mode is generated. t a .

[0093] Temporal encoding of environmental sensor data: One-dimensional convolutional networks are used to process environmental sensor data (such as temperature, humidity, and light intensity). One-dimensional convolutional networks can effectively capture the local features and periodic changes of environmental data over time. In this way, a temporal representation r of the environmental modalities is generated. t e .

[0094] 302. Compress the temporal representation of each mode to generate compressed features for each mode.

[0095] After generating temporal representations for each modality, these temporal representations need to be compressed to reduce feature dimensionality and improve computational efficiency, resulting in more compact feature representations. This process is achieved through a specific compression function. The implementation is as follows: Video modality compression: Temporal representation of video modalities using a sliding window temporal transformer { r t f | t ∈ T Compression is performed to generate compressed features. .

[0096] Audio Modal Compression: Temporal Representation of Audio Modalities Using a Bidirectional GRU Network r t a | t ∈ T Compression is performed to generate compressed features. .

[0097] Environmental mode compression: Temporal representation of environmental modes using one-dimensional convolutional networks { r t e | t ∈ T Compression is performed to generate compressed features. .

[0098] The specific formula is as follows:

[0099] in, Let T represent the compression function for mode k, and let T represent the preset time window. This represents the characteristic representation of mode k at time t.

[0100] 303. Perform cross-modal fusion on the compression features of each modality to generate the fused features.

[0101] After generating compressed features for each modality, these features need to be fused across modalities to generate fused features that can characterize dynamic events within the region. The purpose of cross-modal fusion is to integrate information from different modalities to form a comprehensive feature representation, thereby more comprehensively reflecting the dynamic events occurring within the region.

[0102] Cross-modal fusion method: This invention employs a query vector-based cross-modal fusion method. Specifically, a set of learnable query vectors is introduced and interacts with the compressed features of each modality through a multi-head attention mechanism. This method can dynamically aggregate semantic features from different modalities, capturing the correlation and complementarity between modalities.

[0103] Fusion process: Let They are spatial region nodes. R i The compressed features of video, audio, and environmental modalities are extracted. These compressed features are then fused using a multi-head attention mechanism via the Q-Former module (a lightweight cross-modal fusion module) to generate the final fused feature z. i :

[0104] The Q-Former module dynamically aggregates semantic features through the interaction between query vectors and modal compression features, generating fusion features z that can characterize dynamic events within a region. i .

[0105] Ultimately, fusion features As a spatial region node A unified representation is embedded into the graph structure to obtain the node feature set. The spatial topology map is expanded into a fused map. .

[0106] Taking a living room node in a smart community as an example, suppose that within a preset time window T, this node collects the following multimodal data: Video data: Captures the movement trajectories of people in the living room.

[0107] Audio data: Recorded conversations between people.

[0108] Environmental data: Changes in temperature and light intensity were monitored.

[0109] See Figure 4 The specific steps include: (1) Timing coding.

[0110] Video data is encoded using a sliding window temporal Transformer to generate temporal representations of video modalities. t f .

[0111] Audio data is encoded using a bidirectional GRU to generate a temporal representation r of the audio modalities. t a .

[0112] Environmental data is encoded through a one-dimensional convolutional network to generate a temporal representation r of environmental modalities. t e .

[0113] (2) Compression processing.

[0114] Temporal representation of video modalities {r t f | t ∈ T Compression is performed using a sliding window temporal Transformer to generate compressed features. .

[0115] Temporal representation of audio modes {r t a | t ∈ T Compression is performed using a bidirectional GRU to generate compressed features. .

[0116] Temporal representation of environmental modes {r t e | t ∈ T Compression is achieved through a one-dimensional convolutional network to generate compressed features. .

[0117] (3) Cross-modal fusion.

[0118] Using the Q-Former module, compression features are... Perform cross-modal fusion to generate the final fused feature z. i .

[0119] The fusion feature z generated through the above steps i As attributes of corresponding nodes in the spatial topology graph, they provide rich semantic information for subsequent knowledge graph construction and dynamic event modeling. This process not only captures the temporal dynamics of multimodal data, but also integrates information from different modalities through cross-modal fusion, thus reflecting the dynamic events occurring within the region more comprehensively.

[0120] After constructing the fusion graph that forms the knowledge graph, in order to achieve accurate identification of user intent and behavior prediction, the information in the fusion graph needs to be further processed and input into a pre-trained large language model for semantic reasoning. See also Figure 5 This process includes two key steps: natural language description transformation that integrates graph information and semantic reasoning based on a large language model.

[0121] Step 501: Convert the information in the fused graph into a natural language description.

[0122] After the fusion graph is constructed, its semantic information needs to be converted into natural language descriptions so that it can be input into a large language model for further processing.

[0123] This process specifically includes the following two sub-steps: 1) Apply a graph neural network to the fused graph and enhance the contextual semantics of the fused features of each node through the adjacency propagation mechanism to obtain the enhanced fused features.

[0124] Graph Neural Networks (GNNs) are applied to fusion graphs to enhance the contextual semantic information of the fused features of each node. GNNs use an adjacency propagation mechanism to allow the fused features of a node to interact with the features of its neighbors, thereby capturing the structured relationships and contextual information between nodes. Specifically, the update process of each layer of a GNN is defined as follows:

[0125] in: Indicates that node i is at the lth time. Layer feature representation; Represents the set of neighboring nodes of node i; It is the learnable weight matrix of the l-th layer; It is an activation function, such as ReLU; AGG stands for aggregation function, such as multi-head attention mechanism.

[0126] Through the propagation of multi-layer graph neural networks, the fusion features of nodes are enhanced, which can better reflect their contextual semantic information in the graph structure.

[0127] 2) The spatial topological relationship between the target node and its neighboring nodes, along with the enhanced fusion features, are converted into the natural language description.

[0128] After processing by the graph neural network, the enhanced features of each node not only contain its own semantic information but also incorporate the contextual information of its neighboring nodes. Next, these enhanced features and their corresponding spatial topological relationships need to be converted into natural language descriptions. This process is implemented through a graph structure summarization module, with the following specific steps: Extracting features from the target node and its neighboring nodes: For target node i, extract its enhanced fusion features. and the feature set of its neighboring nodes. .

[0129] Integrate spatial topology relationships: Integrate the spatial topology relationships (such as node type, adjacency relationship, etc.) of the target node and its neighboring nodes with the enhanced fusion features.

[0130] Generate Natural Language Description: The graph structure summarization module converts the above information into a natural language description. For example, for the living room node... R 1 Its natural language description might be as follows: Human activity was detected in the living room, with an ambient temperature of 25°C and a light intensity of 500 lux. Related activity was observed in the kitchen area, which is connected to the living room via a door.

[0131] Step 502: Input the natural language description into a pre-trained large language model for semantic reasoning to generate a structured recognition result or behavior prediction of the user's intent.

[0132] The generated natural language description is input into a pre-trained large language model (such as GPT or BERT), leveraging its powerful semantic understanding capabilities for reasoning. Large language models can parse the semantic information in natural language descriptions, generating structured recognition results or behavioral predictions of user intent. For example: User intent recognition: If the natural language description mentions "someone in the living room is looking for the remote control", the large language model can recognize that the user intent is "looking for the remote control".

[0133] Behavior prediction: If the description mentions "cooking activities in the kitchen and people waiting in the living room", the large language model can predict that the user's likely behavior is "preparing for a meal".

[0134] Since large language models cannot directly process graph structure information, a graph structure summarization module S is introduced to encode the structural state of graph nodes and their neighbors into a natural language description (Prompt).

[0135] in, It is the natural language description of node i. These are fused features enhanced by a graph neural network. L Meta represents the maximum number of layers in a neural network. i It contains the node's metadata.

[0136] To enhance the system's adaptability to complex events and delayed behaviors, a multi-round interactive update mechanism for the graph-LLM is designed. The semantic reasoning results generated by the large language model are injected into the fused graph as new knowledge feedback to dynamically update the attributes of the corresponding nodes.

[0137] Specifically, each round of LLM response This knowledge can be injected into graph nodes as new knowledge and update the graph state for the next round of reasoning. The update process can be described as follows:

[0138] in, The map update method is implemented using a gated control unit (GRU). Response i It is the response generated by the model.

[0139] Graph nodes continuously gain semantic enhancement and state updates through multiple rounds of language interaction, ultimately building an integrated intelligent framework that supports context awareness, structural memory, and behavioral guidance, enabling human-computer collaborative interaction in intelligent spaces.

[0140] To further understand the solutions of the embodiments of the present invention, Figure 6 This diagram illustrates a flowchart of the scalable smart community knowledge graph construction method of this embodiment. This method enhances the perception and reasoning capabilities of intelligent spaces by fusing multimodal data and utilizing graph neural networks. It includes: S1: Multimodal data acquisition and preprocessing.

[0141] Data is collected from various sensors in the smart space, including video, audio, and environmental sensor data.

[0142] The collected multimodal data is preprocessed, including data cleaning, format standardization, and preliminary feature extraction.

[0143] S2: Physical space topology construction.

[0144] Based on the physical layout of the smart community, a spatial topology is constructed, defining each spatial region (node) and its physical adjacency relationship (edge).

[0145] Map the sensors to the corresponding spatial area nodes according to their deployment locations, and add node attributes.

[0146] S3: Feature extraction from pre-trained encoder.

[0147] Feature extraction is performed on the pre-processed multimodal data using a pre-trained encoder.

[0148] Single-moment feature compression representation is performed on video data, and the same operation is performed on audio and environmental data.

[0149] S4: Enhanced graph neural network structure.

[0150] Graph neural networks are applied to enhance the contextual semantics of node fusion features, updating node features through an adjacency propagation mechanism. Natural language descriptions are generated using the output of the graph neural network, integrating the modal states of nodes and the contextual events of neighboring nodes.

[0151] The node features enhanced by the graph neural network and the natural language description are input into a pre-trained Large Language Model (LLM) for semantic reasoning. The LLM automatically infers the current interaction state based on the input description, identifies the user's intent, and generates structured response instructions or prediction results.

[0152] The semantic reasoning results generated by LLM are injected into the graph nodes as new knowledge feedback, and the graph state is updated for the next round of reasoning. Furthermore, methods such as Gated Execution Units (GRUs) are used to achieve dynamic updates of the graph state to adapt to new information and environmental changes.

[0153] Through the steps described above, the knowledge graph of the intelligent space is continuously updated and improved to reflect dynamic events and user behaviors occurring within the community. This approach not only enhances the intelligent space's ability to understand and respond to user intentions but also strengthens the system's adaptability and reasoning capabilities in the face of complex events.

[0154] The following describes the scalable smart community knowledge graph construction system provided by the embodiments of the present invention. The scalable smart community knowledge graph construction system described below can be referred to in correspondence with the scalable smart community knowledge graph construction method described above.

[0155] This invention provides a scalable smart community knowledge graph construction system, see [link / reference]. Figure 7 ,include: The preprocessing module 710 is used to preprocess the multimodal sensing data collected from multiple spatial areas within the smart community; The topology graph construction module 720 is used to construct a spatial topology graph with the spatial regions as nodes and the physical adjacency relationships as edges, based on multiple preset spatial regions within the smart community and their physical adjacency relationships. The knowledge graph construction module 730 is used to process the preprocessed multimodal perception data to generate fusion features that characterize dynamic events in the region, and to use the fusion features as attributes of corresponding nodes in the spatial topology graph to construct the fusion graph of the knowledge graph.

[0156] Figure 8 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 8As shown, the electronic device may include a processor 810, a communications interface 820, a memory 830, and a communication bus 840, wherein the processor 810, communications interface 820, and memory 830 communicate with each other via the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute a scalable smart community knowledge graph construction method. This method includes: preprocessing multimodal sensing data collected from multiple spatial regions within the smart community; constructing a spatial topology graph with the spatial regions as nodes and the physical adjacencies as edges based on multiple preset spatial regions and their physical adjacencies within the smart community; processing the preprocessed multimodal sensing data to generate fusion features representing dynamic events within the region, and using the fusion features as attributes of corresponding nodes in the spatial topology graph to construct a fusion graph of the knowledge graph.

[0157] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0158] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the scalable smart community knowledge graph construction method provided by the above methods. The method includes: preprocessing multimodal sensing data collected from multiple spatial regions within the smart community; constructing a spatial topology graph with the spatial regions as nodes and the physical adjacency relationships as edges based on multiple preset spatial regions within the smart community and their physical adjacency relationships; processing the preprocessed multimodal sensing data to generate fusion features representing dynamic events within the region, and using the fusion features as attributes of corresponding nodes in the spatial topology graph to construct a fusion graph of the knowledge graph.

[0159] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements a method for constructing a scalable smart community knowledge graph provided by the methods described above. The method includes: preprocessing multimodal sensing data collected from multiple spatial regions within the smart community; constructing a spatial topology graph with the spatial regions as nodes and the physical adjacency relationships as edges based on multiple preset spatial regions within the smart community and their physical adjacency relationships; processing the preprocessed multimodal sensing data to generate fusion features representing dynamic events within the region, and using the fusion features as attributes of corresponding nodes in the spatial topology graph to construct a fusion graph of the knowledge graph.

[0160] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0161] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0162] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A scalable method for constructing a knowledge graph for smart communities, characterized in that, include: Preprocessing of multimodal sensing data collected from multiple spatial areas within the smart community; Based on the multiple spatial regions and their physical adjacency relationships within the smart community, a spatial topology graph is constructed with the spatial regions as nodes and the physical adjacency relationships as edges. The preprocessed multimodal sensing data is processed to generate fusion features that characterize dynamic events in the region, and the fusion features are used as attributes of corresponding nodes in the spatial topology graph to construct the fusion graph of the knowledge graph.

2. The method according to claim 1, characterized in that, The multimodal sensing data includes at least one of video data, audio data, and environmental sensor data; Preprocessing of multimodal sensing data collected from multiple spatial areas within the smart community specifically includes: When the multimodal sensing data includes video data, the video stream is uniformly sampled and normalized; When the multimodal sensing data includes audio data, the audio data is denoised and normalized. When the multimodal sensing data includes environmental sensor data, a low-pass filter is used to filter, denoise, and normalize the environmental sensor data.

3. The method according to claim 1, characterized in that, Based on multiple pre-defined spatial regions within the smart community and their physical adjacency relationships, a spatial topology graph is constructed, with the spatial regions as nodes and the physical adjacency relationships as edges. Specifically, this includes: The spatial areas are divided according to the physical layout of the smart community, serving as nodes in the spatial topology map; The sensors deployed in each spatial area are assigned to the nodes based on their physical locations; Physical adjacency relationships between nodes are established based on whether there are physical connections between different spatial regions, so that the nodes and the physical adjacency relationships together construct the spatial topology graph.

4. The method according to claim 1, characterized in that, The preprocessed multimodal sensing data is then processed to generate fusion features characterizing dynamic events within the region, specifically including: The multimodal sensing data is temporally encoded within a preset time window to generate temporal representations of each modality. The temporal representations of each mode are compressed to generate compressed features for each mode; The compressed features of each modality are fused across modalities to generate the fused features; the cross-modal fusion is to use a set of learnable query vectors to generate the fused features through cross-attention interaction aggregation with the compressed features of each modality.

5. The method according to claim 1, characterized in that, After constructing the fusion graph that forms the knowledge graph, the method further includes: Convert the information in the fused graph into a natural language description; The natural language description is input into a pre-trained large language model for semantic reasoning to generate a structured recognition result or behavior prediction of the user's intent.

6. The method according to claim 5, characterized in that, Converting the information in the fused graph into a natural language description specifically includes: A graph neural network is applied to the fused graph, and the contextual semantics of the fused features of each node are enhanced through the adjacency propagation mechanism to obtain the enhanced fused features; The spatial topological relationship between the target node and its neighboring nodes, along with the enhanced fusion features, are combined and converted into the natural language description.

7. The method according to claim 5 or 6, characterized in that, The method further includes: The semantic reasoning results generated by the large language model are injected into the fusion graph as new knowledge feedback to dynamically update the attributes of the corresponding nodes.

8. A scalable smart community knowledge graph construction system, characterized in that, include: The preprocessing module is used to preprocess the multimodal sensing data collected from multiple spatial areas within the smart community. The topology graph construction module is used to construct a spatial topology graph with the spatial regions as nodes and the physical adjacency relationships as edges, based on multiple preset spatial regions within the smart community and their physical adjacency relationships. The knowledge graph construction module is used to process the preprocessed multimodal perception data to generate fusion features that characterize dynamic events in the region, and to use the fusion features as attributes of corresponding nodes in the spatial topology graph to construct the fusion graph of the knowledge graph.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the scalable smart community knowledge graph construction method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the scalable smart community knowledge graph construction method as described in any one of claims 1 to 7.

Images