Method and device for tracing a family event, electronic device and storage medium

Abstract

Embodiments of the present application provide a household event tracing method and device, electronic equipment and storage medium, relating to the technical field of Internet of Things, the method comprises: obtaining an abnormal event corresponding to an event query instruction, and a causal knowledge graph corresponding to a home environment, the causal knowledge graph comprises a plurality of atomic events and a causal cohesion score corresponding to adjacent two atomic events, then searching a candidate causal chain corresponding to the abnormal event from the causal knowledge graph, and finally screening a target causal chain for the event query instruction from the candidate causal chain according to the causal cohesion score, and outputting event description information corresponding to the target causal chain, thereby realizing efficient and accurate complex event tracing, restoring the corresponding household event for the user, and ensuring privacy, while realizing real-time and reliable decision support, which can respond to user queries in real time, output reliable event descriptions, and greatly improve the usability of information and user experience.

Description

Technical Field

[0001] This invention relates to the field of Internet of Things (IoT) technology, and in particular to a method for tracing household events, a device for tracing household events, an electronic device, and a computer-readable storage medium. Background Technology

[0002] With the development of IoT and security technologies, users can deploy one or more network cameras in their home environment to record video continuously or based on motion detection. In the event of theft, accidents, or other incidents, users can review the video recordings to intuitively review what happened. However, cameras indiscriminately record all activities of family members in their private spaces, including living, dressing, and leisure activities, leaving no room for personal privacy and dignity. This "trading privacy for truth" model is psychologically and ethically unacceptable to family members, limiting its application to quasi-public areas such as doorways and courtyards, making it unsuitable for resolving everyday family events. Furthermore, the log information collected by multiple smart devices is atomized, discrete, and distributed across different data silos, failing to construct complete causal chains between events and lacking the ability to comprehensively analyze and trace the origins of complex, multi-factor-related family events. Summary of the Invention

[0003] The present invention provides a method, apparatus, electronic device, and computer-readable storage medium for tracing family events, in order to solve or partially solve the problems of poor privacy and lack of comprehensive analysis and tracing capabilities for complex family events involving multiple factors when tracing family events in the home environment.

[0004] This invention discloses a method for tracing the source of family events, including:

[0005] In response to an event query command for a home environment, the system obtains the abnormal event corresponding to the event query command and the causal knowledge graph corresponding to the home environment. The causal knowledge graph includes several atomic events and causal cohesion scores corresponding to two adjacent atomic events.

[0006] Search the causal knowledge graph for candidate causal chains corresponding to the anomalous event;

[0007] Based on the causal cohesion score, the target causal chain for the event query instruction is selected from the candidate causal chains, and the event description information corresponding to the target causal chain is output.

[0008] In some feasible implementations, the candidate causal chain includes several target atomic events, and the step of filtering the target causal chain for the event query instruction from the candidate causal chains based on the causal cohesion score includes:

[0009] Obtain the target causal cohesion score between two adjacent target atomic events in the same candidate causal chain, and calculate the credibility of the candidate causal chain based on the target causal cohesion score;

[0010] The candidate causal chain with the highest credibility is selected as the target causal chain for the event query command.

[0011] Among some feasible implementation methods are:

[0012] In response to a family event occurring in the family environment, target data associated with the family event is acquired, and the target data is analyzed to obtain the corresponding atomic events;

[0013] Obtain the event information corresponding to each of the atomic events;

[0014] Calculate the causal cohesion score between the two atomic events based on the event information;

[0015] Create nodes corresponding to the atomic events, and use the causal cohesion scores as relation edges to build a causal knowledge graph corresponding to the family environment based on each atomic event.

[0016] In some feasible implementations, the target data includes at least radar signals, the atomic events include at least behavioral events, and the analysis of the target data to obtain the corresponding atomic events includes:

[0017] The radar signal is segmented into frames to obtain the corresponding signal frames;

[0018] Perform a short-time Fourier transform on the signal frame to obtain the corresponding transform result;

[0019] The transformation results corresponding to the consecutive signal frames are arranged in chronological order to obtain the micro-Doppler spectrum of the radar signal.

[0020] The micro-Doppler spectrogram is identified to obtain multiple behavioral labels and a first confidence score;

[0021] The behavior label and the first confidence score are encapsulated to obtain the behavior event corresponding to the radar signal.

[0022] In some feasible implementations, the target data includes at least an audio signal, the atomic event includes at least an audio event, and the analysis of the target data to obtain the corresponding atomic events includes:

[0023] The sound signal is divided into frames to obtain the corresponding audio frames;

[0024] Perform a Fourier transform on the audio frame to obtain the corresponding spectrum;

[0025] The spectrum is converted into the corresponding Mel scale, and the Mel scale corresponding to consecutive audio frames is arranged in chronological order to obtain the corresponding Mel spectrogram;

[0026] Feature extraction is performed on the Mel spectrogram to obtain multiple first sound labels and second confidence scores corresponding to the first sound labels, multiple second sound labels and third confidence scores corresponding to the second sound labels;

[0027] The first sound tag and the second confidence score are encapsulated to obtain the first sound event corresponding to the sound signal, and the second sound tag and the third confidence score are encapsulated to obtain the second sound event corresponding to the sound signal.

[0028] In some feasible implementations, the target data includes at least a touch signal, the atomic event includes at least a touch event, and the analysis of the target data to obtain the corresponding atomic event includes:

[0029] Obtain the touch parameters and the fourth confidence score corresponding to the touch signal;

[0030] Each of the touch parameters is used as a touch tag and encapsulated with the fourth confidence score to obtain the touch event corresponding to the touch signal.

[0031] In some feasible implementations, the target data includes at least a vibration signal, the atomic event includes at least a vibration event, and the analysis of the target data to obtain the corresponding atomic event includes:

[0032] If the duration of the vibration signal within the target time window reaches a preset threshold, then the vibration parameters and the fifth confidence score corresponding to the vibration signal are obtained.

[0033] Each vibration parameter is used as a vibration label and encapsulated with the fifth confidence score to obtain the vibration event corresponding to the vibration signal.

[0034] In some feasible implementations, the event information includes at least the trigger time and trigger location, the atomic events include at least a first atomic event and a second atomic event, and the calculation of the causal cohesion score between the two atomic events based on the event information includes:

[0035] The spatiotemporal cohesion between the first atomic event and the second atomic event is calculated by using the first trigger time and the first trigger position of the first atomic event and the second trigger time and the second trigger position of the second atomic event.

[0036] Obtain the prior causal potential and data-driven correlation between the first atomic event and the second atomic event, and use the prior causal potential and the data-driven correlation to calculate the causal cohesion between the first atomic event and the second atomic event.

[0037] Obtain the confidence level of the result of the second atomic event relative to the first atomic event;

[0038] The causal cohesion score between the first atomic event and the second atomic event in the same candidate causal chain is obtained by using the spatiotemporal cohesion, the causal cohesion, and the result confidence score.

[0039] In some feasible implementations, the step of calculating the spatiotemporal cohesion between the first atomic event and the second atomic event by using the first trigger time and first trigger position of the first atomic event and the second trigger time and second trigger position of the second atomic event includes:

[0040] The time difference is calculated by using the first trigger time of the first atomic event and the second trigger time of the second atomic event;

[0041] The spatial distance is calculated using the first trigger position of the first atomic event and the second trigger position of the second atomic event;

[0042] The spatiotemporal cohesion between the first atomic event and the second atomic event is obtained by calculating the time difference and the spatial distance.

[0043] This invention also discloses a device for tracing family events, comprising:

[0044] The response module is used to respond to an event query command for the home environment, obtain the abnormal event corresponding to the event query command, and the causal knowledge graph corresponding to the home environment. The causal knowledge graph includes a number of atomic events and causal cohesion scores corresponding to two adjacent atomic events.

[0045] The search module is used to search for candidate causal chains corresponding to the abnormal event from the causal knowledge graph.

[0046] The feedback module is used to filter out the target causal chain for the event query instruction from the candidate causal chains based on the causal cohesion score, and output the event description information corresponding to the target causal chain.

[0047] In some feasible implementations, the candidate causal chain includes several target atomic events, and the feedback module is specifically used for:

[0048] Obtain the target causal cohesion score between two adjacent target atomic events in the same candidate causal chain, and calculate the credibility of the candidate causal chain based on the target causal cohesion score;

[0049] The candidate causal chain with the highest credibility is selected as the target causal chain for the event query command.

[0050] Among some feasible implementation methods are:

[0051] An event construction module is used to respond to a family event occurring in the family environment, acquire target data associated with the family event, and analyze the target data to obtain the corresponding atomic events;

[0052] The information acquisition module is used to acquire event information corresponding to each of the atomic events;

[0053] The calculation module is used to calculate the causal cohesion score between two atomic events based on the event information;

[0054] The graph construction module is used to create nodes corresponding to the atomic events and use the causal cohesion scores as relation edges to build a causal knowledge graph corresponding to the family environment based on each atomic event.

[0055] In some feasible implementations, the target data includes at least radar signals, the atomic events include at least behavioral events, and the event construction module is specifically used for:

[0056] The radar signal is segmented into frames to obtain the corresponding signal frames;

[0057] Perform a short-time Fourier transform on the signal frame to obtain the corresponding transform result;

[0058] The transformation results corresponding to the consecutive signal frames are arranged in chronological order to obtain the micro-Doppler spectrum of the radar signal.

[0059] The micro-Doppler spectrogram is identified to obtain multiple behavioral labels and a first confidence score;

[0060] The behavior label and the first confidence score are encapsulated to obtain the behavior event corresponding to the radar signal.

[0061] In some feasible implementations, the target data includes at least a sound signal, the atomic event includes at least a sound event, and the event construction module is specifically used for:

[0062] The sound signal is divided into frames to obtain the corresponding audio frames;

[0063] Perform a Fourier transform on the audio frame to obtain the corresponding spectrum;

[0064] The spectrum is converted into the corresponding Mel scale, and the Mel scale corresponding to consecutive audio frames is arranged in chronological order to obtain the corresponding Mel spectrogram;

[0065] Feature extraction is performed on the Mel spectrogram to obtain multiple first sound labels and second confidence scores corresponding to the first sound labels, multiple second sound labels and third confidence scores corresponding to the second sound labels;

[0066] The first sound tag and the second confidence score are encapsulated to obtain the first sound event corresponding to the sound signal, and the second sound tag and the third confidence score are encapsulated to obtain the second sound event corresponding to the sound signal.

[0067] In some feasible implementations, the target data includes at least a touch signal, the atomic event includes at least a touch event, and the event construction module is specifically used for:

[0068] Obtain the touch parameters and the fourth confidence score corresponding to the touch signal;

[0069] Each of the touch parameters is used as a touch tag and encapsulated with the fourth confidence score to obtain the touch event corresponding to the touch signal.

[0070] In some feasible implementations, the target data includes at least a vibration signal, the atomic event includes at least a vibration event, and the event construction module is specifically used for:

[0071] If the duration of the vibration signal within the target time window reaches a preset threshold, then the vibration parameters and the fifth confidence score corresponding to the vibration signal are obtained.

[0072] Each vibration parameter is used as a vibration label and encapsulated with the fifth confidence score to obtain the vibration event corresponding to the vibration signal.

[0073] In some feasible implementations, the event information includes at least the trigger time and trigger location, the atomic events include at least a first atomic event and a second atomic event, and the calculation module is specifically used for:

[0074] The spatiotemporal cohesion between the first atomic event and the second atomic event is calculated by using the first trigger time and the first trigger position of the first atomic event and the second trigger time and the second trigger position of the second atomic event.

[0075] Obtain the prior causal potential and data-driven correlation between the first atomic event and the second atomic event, and use the prior causal potential and the data-driven correlation to calculate the causal cohesion between the first atomic event and the second atomic event.

[0076] Obtain the confidence level of the result of the second atomic event relative to the first atomic event;

[0077] The causal cohesion score between the first atomic event and the second atomic event in the same candidate causal chain is obtained by using the spatiotemporal cohesion, the causal cohesion, and the result confidence score.

[0078] In some feasible implementations, the computing module is specifically used for:

[0079] The time difference is calculated by using the first trigger time of the first atomic event and the second trigger time of the second atomic event;

[0080] The spatial distance is calculated using the first trigger position of the first atomic event and the second trigger position of the second atomic event;

[0081] The spatiotemporal cohesion between the first atomic event and the second atomic event is obtained by calculating the time difference and the spatial distance.

[0082] This invention also discloses an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0083] The memory is used to store computer programs;

[0084] When the processor executes a program stored in the memory, it implements the method described in the embodiments of the present invention.

[0085] This invention also discloses a computer-readable storage medium storing instructions that, when executed by one or more processors, cause the processors to perform the methods described in this invention.

[0086] The embodiments of the present invention have the following advantages:

[0087] In this embodiment of the invention, when a user wants to query the event description corresponding to a family event that has already occurred in the home environment, such as the cause and time of the event, they can input the corresponding event query command. Then, by obtaining the abnormal event corresponding to the event query command and the causal knowledge graph corresponding to the home environment, which includes several atomic events and the causal cohesion scores corresponding to two adjacent atomic events, the system searches for candidate causal chains corresponding to the abnormal event in the causal knowledge graph. Finally, based on the causal cohesion scores, the system selects the target causal chain for the event query command from the candidate causal chains and outputs the event description information corresponding to the target causal chain. Thus, by constructing a causal knowledge graph and using the pre-calculated causal cohesion scores for reasoning, the system can achieve efficient and accurate tracing of complex events, reconstruct the corresponding family events for the user, and ensure privacy. At the same time, it achieves real-time and reliable decision support, can respond to user queries in real time, output reliable event descriptions, and further output humanized and understandable event description information, greatly improving the usability of information and user experience. Attached Figure Description

[0088] Figure 1 This is a flowchart of the steps of a family event tracing method provided in an embodiment of the present invention;

[0089] Figure 2 This is a schematic diagram of the event tracing process provided in the embodiments of the present invention;

[0090] Figure 3 This is a structural block diagram of a family event tracing device provided in an embodiment of the present invention. Detailed Implementation

[0091] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0092] As an example, in a home environment, users can deploy one or more network cameras to record continuous or motion-detection-based videos. In the event of theft, accidents, or other incidents, users can review the video recordings to visually review the events, severely compromising their privacy and security. Furthermore, individual smart devices, such as smart locks, smart curtains, and smart sockets, typically record their own operation logs, allowing users to view information such as "the door lock was opened by the administrator at 3:10 PM" or "the living room socket was powered off due to overload protection at 8:00 PM" within their respective branded applications. However, the information recorded by these individual smart devices is isolated and completely lacks causal correlation. The logs recorded by different smart devices are atomized, discrete, and distributed across different data silos, failing to form a coherent, contextualized event storyline. Furthermore, users cannot know who opened the door lock, nor can they know whether the power outage was due to "connecting a specific high-power appliance" or "unstable grid voltage." Smart devices only record "what happened," but cannot answer "who," "why," or "how," and cannot construct a complete causal chain between events. They lack the ability to comprehensively analyze and trace the source of complex, multi-factor-related family events.

[0093] In this invention, for family events that have already occurred in the home environment, when a user wants to query the event description corresponding to the corresponding family event, such as the cause and time of the event, they can input the corresponding event query command. Then, by obtaining the abnormal event corresponding to the event query command and the causal knowledge graph corresponding to the family environment, which includes several atomic events and the causal cohesion scores corresponding to two adjacent atomic events, the causal knowledge graph searches for candidate causal chains corresponding to the abnormal event. Finally, based on the causal cohesion scores, the target causal chain corresponding to the event query command is selected from the candidate causal chains, and the event description information corresponding to the target causal chain is output. Thus, by constructing a causal knowledge graph and using the pre-calculated causal cohesion scores for reasoning, efficient and accurate tracing of complex events can be achieved, restoring the corresponding family events for the user while ensuring privacy. At the same time, real-time and reliable decision support is achieved, which can respond to user queries in real time, output reliable event descriptions, and further output humanized and understandable event description information, greatly improving the usability of information and user experience.

[0094] To enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, some technical features in the embodiments of the present invention are explained and described below:

[0095] An atomic event is the smallest indivisible unit of an event captured by a single sensor or a single signal processing operation. It forms the basis for more complex events.

[0096] Structural Causal Model (SCM): A directed acyclic graph (DAG) containing variables and their direct causal relationships (represented by directed edges), used as a prior knowledge base for the inference engine.

[0097] Causal Knowledge Graph (CKG): A dynamic data structure that uses graph database technology to store all atomic events and their complex temporal, spatial, and causal relationships in real time.

[0098] Spatio-Tempora and Causal Cohesion (STC) 2 ): A scoring model that scores the causal relationship between two atomic events, used to quantitatively assess the credibility of a sequence of events forming a coherent causal story.

[0099] Reference Figure 1 The diagram illustrates a flowchart of a method for tracing the origin of family events provided in an embodiment of the present invention, which may specifically include the following steps:

[0100] Step 101: In response to an event query command for the family environment, obtain the abnormal event corresponding to the event query command and the causal knowledge graph corresponding to the family environment. The causal knowledge graph includes a number of atomic events and causal cohesion scores corresponding to two adjacent atomic events.

[0101] In this embodiment of the invention, a corresponding smart home system can be deployed in the home environment. In addition to providing users with daily smart home services, the smart home system can also perform "non-visual" perception of home events. When a corresponding home event is detected, the corresponding data can be collected and atomic events corresponding to the home event can be generated to construct a causal knowledge graph corresponding to the home environment. This makes it convenient for users to input corresponding query commands to query the home events that have occurred in the home environment.

[0102] Optionally, a smart home system can include at least a data perception layer, an edge computing layer, and a user interaction layer. The data perception layer can act as the system's "sensors," consisting of a series of non-visual sensors authorized by the user, such as millimeter-wave radar for sensing spatial dynamics, microphone arrays for recognizing sound signals, piezoelectric vibration sensors and door / window contact sensors for detecting physical contact, and smart home device APIs (Application Programming Interfaces) for acquiring device status. The edge computing layer can act as the system's "brain," a server deployed in the user's home network where all data from the perception layer can be processed. It can include at least a signal processing engine, a pattern recognition model library, a causal inference engine, a CKG graph database, and a locally deployed large language model interface, thus ensuring data remains within the home and eliminating the risk of cloud data leakage. The user interaction layer can be an application installed on the user's smartphone, tablet, or smart home screen, providing a natural language dialogue-based interactive interface through which users can initiate queries and receive illustrated traceability reports.

[0103] In one example, when a corresponding family event occurs in the home environment, the data perception layer is responsible for collecting raw data associated with the event and transmitting the raw data to the edge computing layer. The edge computing layer then infers from the raw data and constructs a corresponding causal knowledge graph. When the user inputs a corresponding query command through the user interaction layer, the edge computing layer can return the corresponding query results based on the causal knowledge graph. This provides the user with a safe, reliable, and intelligent "black box" for family events, eliminating the need for users to deploy cameras that could easily infringe on their privacy in the home environment, thus ensuring the security of user privacy. It also eliminates the need to futilely search for clues in complex and isolated device logs. Users can quickly and clearly understand various family events that occur in their home through simple natural language dialogue, greatly improving the convenience and flexibility of event tracing.

[0104] For users, when they want to query what kind of family events have occurred in their home environment or what caused them, they can input the corresponding event query command through the user interaction layer. The system responds to the user's input of the event query command for the home environment, obtains the abnormal event corresponding to the event query command, and the causal knowledge graph corresponding to the home environment, so as to query the event information corresponding to the event query command through the causal knowledge graph.

[0105] The causal knowledge graph can include several atomic events and the corresponding causal cohesion scores between two adjacent atomic events. The causal cohesion scores can be used to measure the credibility of the causal relationship between two atomic events. The scores can be between 0 and 1. The larger the score, the more obvious the causal relationship between the two atomic events. Conversely, the smaller the score, the less likely there is a causal relationship between the two atomic events.

[0106] In some feasible implementations, a causal knowledge graph can be the content continuously updated by a smart home system. This causal knowledge graph can include several nodes, each corresponding to an atomic event. Atomic events can include Events, Agents (e.g., human_01, pet_01), and Objects (e.g., vase, door). There are corresponding relationship edges between two atomic events, which can represent the relationship between them. These edges can include relationships such as precedes, located_at, co_occurs_at, and most importantly, likely_causes. The causal knowledge graph can be trained on massive amounts of historical event data using the inductive learning capabilities of a GNN (Graph Neural Network) model. This allows it to learn "general patterns" of causal connections between different types of nodes, enabling the generation of implicit causal relationships specific to the home environment that are not included in the SCM (Smart Home Graph), thus allowing the graph to continuously evolve and improve itself.

[0107] In practical implementation, the smart home system can respond to home events that occur in the home environment, obtain target data associated with the home events, analyze the target data to obtain the corresponding atomic events, then obtain the event information corresponding to each atomic event, calculate the causal cohesion score between two atomic events based on the event information, and finally create nodes corresponding to the atomic events, and use the causal cohesion score as the relation edge to build a causal knowledge graph corresponding to the home environment based on each atomic event.

[0108] The target data can include data collected by different sensors, such as radar signals collected by millimeter-wave radar, sound signals collected by microphone arrays, vibration signals collected by vibration sensors, and touch signals collected by touch sensors. Based on different data, different atomic events can be analyzed, such as behavioral events, sound events, vibration events, and touch events. After the data perception layer collects the corresponding data, it can transmit the collected data to the edge computing layer for analysis. This allows the edge computing layer to convert the raw, continuous, and unstructured sensor data stream into discrete, semantically rich, and structured atomic events, providing a good data foundation for subsequent causal reasoning of family events.

[0109] In some examples, for radar signals, the edge computing layer can perform frame processing on the radar signals to obtain the corresponding signal frames. Then, it can perform short-time Fourier transform on the signal frames to obtain the corresponding transform results. Then, it can arrange the transform results corresponding to consecutive signal frames in chronological order to obtain the micro-Doppler spectrum map corresponding to the radar signal. Then, it can identify the micro-Doppler spectrum map to obtain multiple behavior labels and a first confidence score. Finally, it can encapsulate the behavior labels and the first confidence score to obtain the behavior event corresponding to the radar signal.

[0110] For example, the data sensing layer can use a 60GHz millimeter-wave radar module to continuously transmit frequency-modulated continuous waves. The received echo signal is then mixed with the transmitted signal to obtain an I / Q intermediate frequency signal containing target range, velocity, and angle information—the original signal. This original signal is first processed by a digital filter to remove static environmental clutter (such as reflections from walls and furniture) and background noise. Next, the continuously received I / Q signal is framed, with each frame representing a very short time slice (e.g., 20 milliseconds). Then, a Short-Time Fourier Transform (STFT) is applied to each frame. The core function of the STFT is to calculate the energy distribution of all echo signals at different Doppler frequency shifts within this extremely short time slice. Furthermore, the STFT results of the continuous time frames can be arranged chronologically to generate a two-dimensional time-frequency spectrum, i.e., a micro-Doppler spectrogram (where the X-axis represents time, the Y-axis represents Doppler frequency, and the brightness of a pixel represents the signal energy at that frequency). Because micro-Doppler spectrograms are not random noise, but rather "fingerprints" of specific dynamic behaviors—for example, when a human walks, their torso produces a central Doppler frequency shift, while the limbs swinging back and forth produce periodic positive and negative minute frequency shifts on both sides of this central frequency, forming a unique, periodic sinusoidal texture on the graph—the micro-Doppler spectrogram shows a bright line starting from zero frequency with a gradually increasing slope, and the micro-Doppler spectrogram of a pet, due to its different gait and size, exhibits frequency and periodic characteristics completely different from those of a human. Therefore, the system treats the spectrogram as an "image" and inputs it into a pre-trained convolutional neural network (CNN) model (such as MobileNetV2). Through its convolutional and pooling layers, the CNN can automatically learn and extract the most discriminative texture and morphological features from these spectrograms. The final output layer of the CNN (such as the Softmax layer) can provide a corresponding classification result, that is, the most likely label of the current dynamic event (such as human_walk, pet_run, object_fall, human_fall, etc.), along with a confidence score. Then, based on the generated label and confidence score, a structured atomic event corresponding to the radar signal is constructed. The classification result of the CNN can be used to encapsulate a standardized JSON object, namely an "atomic event".

[0111] Taking the incident of a "broken vase" as an example, when a pet in the home environment runs towards the vase, the corresponding behavioral event generated based on radar signals can be as follows:

[0112] {

[0113] "eventId":"evt_mmw_001",

[0114] "timestamp":"2024-10-26T15:30:01.250Z",

[0115] "source":"Radar_LivingRoom_01",

[0116] "eventType":"DynamicActivity",

[0117] "data":{

[0118] "classLabel":"pet_run",

[0119] "confidence":0.96,

[0120] "estimatedVelocity":1.8, / / m / s

[0121] "direction":[-0.8,0.6,0.0] / / Vector pointing in the direction of the vase

[0122] }

[0123] }

[0124] For this structured behavioral event, the time, place, subject (what it is), behavior (how it is) and corresponding confidence level (e.g., 0.96) are clearly defined, thus enabling the construction of an effective node in the causal knowledge graph.

[0125] For audio signals, the edge computing layer can segment the audio signal into frames to obtain the corresponding audio frames. Then, it can perform Fourier transform on the audio frames to obtain the corresponding spectrum. The spectrum is then converted into the corresponding Mel scale, and the Mel scale corresponding to consecutive audio frames is arranged in chronological order to obtain the corresponding Mel spectrogram. Feature extraction is then performed on the Mel spectrogram to obtain multiple first sound labels and their corresponding second confidence scores, multiple second sound labels and their corresponding third confidence scores. Finally, the first sound labels and second confidence scores are encapsulated to obtain the first sound event corresponding to the audio signal, and the second sound labels and third confidence scores are encapsulated to obtain the second sound event corresponding to the audio signal.

[0126] For example, after the microphone array captures the corresponding ambient sound, the captured sound signal can be subjected to noise reduction and optional sound source localization processing (such as determining the direction of the sound source through beamforming). Then, based on the preprocessed sound signal, it is converted into a corresponding Mel spectrogram. Specifically, the audio waveform is first divided into frames. A Fourier transform is applied to each frame to obtain its spectrum. Then, the spectrum on the linear frequency scale is converted into a Mel scale by passing it through a Mel filter bank. Finally, the Mel spectrograms of consecutive frames are arranged in chronological order to generate a Mel spectrogram, which can be a two-dimensional representation of time and frequency. Furthermore, a convolutional recurrent neural network (CNN) model can be used to extract features and perform temporal analysis on the Mel spectrogram. First, a multi-layer CNN is used to learn and extract local high-level features of the sound in the frequency domain. For example, the sound of "breaking glass" has unique instantaneous impact features covering the high-frequency region; the sound of "flowing water" is a steady broadband noise; and the sound of "footsteps" is a series of low-frequency, rhythmic energy blocks. Then, the feature sequence extracted by the CNN (each feature represents a time step) is fed into a Long Short-Term Memory (LSTM) layer (i.e., a recurrent layer). The LSTM can effectively capture long-term dependencies in the sequence data, identify the sound type, and "understand" the state of the sound based on the intensity changes and time intervals. For example, if it is identified as "footsteps," it can be understood as "footsteps approaching from a distance and getting faster and faster," thus achieving a leap from "sound classification" to "acoustic scene understanding."

[0127] Taking a "kitchen leak" as an example, assuming it's a dishwasher malfunction, the sound signal processing flow might sequentially generate the following two key atomic events:

[0128] First, when the fault occurred, the CRNN (Convolutional Recurrent Neural Network) model detected the abnormal pump operating sound:

[0129] {

[0130] "eventId":"evt_aud_008",

[0131] "timestamp":"2024-10-27T09:15:40.100Z",

[0132] "source":"MicArray_Kitchen_01",

[0133] "eventType":"AcousticScene",

[0134] "data":{

[0135] "classLabel":"pump_noise_abnormal",

[0136] "confidence":0.91,

[0137] "description":"Sound signature deviates from normal dishwasher pumpoperation.",

[0138] "intensity": 65 / / dB

[0139] }

[0140] Secondly, a continuous sound of flowing water can also be detected:

[0141] {

[0142] "eventId":"evt_aud_009",

[0143] "timestamp":"2024-10-27T09:15:42.500Z",

[0144] "source":"MicArray_Kitchen_01",

[0145] "eventType":"AcousticScene",

[0146] "data":{

[0147] "classLabel":"water_detected",

[0148] "confidence":0.97,

[0149] "description":"Continuous water flow sound detected.",

[0150] "intensity":58 / / dB

[0151] }

[0152] }

[0153] Thus, by converting unstructured data into structured atomic events, decisive, data-driven evidence can be provided for subsequent scoring models to determine whether "abnormal pumping noise" is caused by "water leakage" or equipment failure.

[0154] It's important to note that in the processing of sound signals, the results of sound event processing can include two levels: basic classification labels and structured scene descriptions. Basic classification labels (such as "sound words") can be isolated, sequential events. A smart home system can output a simple classification label, similar to behavior labels, such as: glass_breaking, water_detected, door_slam, etc. For structured scene descriptions (such as "sound stories"), the recurrent layers in a convolutional recurrent neural network can effectively capture the dynamic changes and context of the sound, thus outputting a structured object containing multi-dimensional information, such as for footsteps:

[0155] A simple tag is "footsteps_detected", while a structured scene description can be:

[0156] {

[0157] "event":"footsteps",

[0158] "direction":"approaching", / / Direction: from far to near

[0159] "tempo":"accelerating" / / tempo: getting faster and faster

[0160] }

[0161] Through the above process, not only can the corresponding sound type be determined, but the state of the sound can also be described, such as "footsteps" or "footsteps approaching from a distance." This invention does not impose any limitations on this.

[0162] For touch signals, the system can obtain the touch parameters and the fourth confidence score corresponding to the touch signal, and then encapsulate each touch parameter as a touch label with the fourth confidence score to obtain the touch event corresponding to the touch signal.

[0163] For vibration signals, if the duration of the vibration signal within the target time window reaches a preset threshold, the vibration parameters and the fifth confidence score corresponding to the vibration signal are obtained. Then, each vibration parameter is used as a vibration label and encapsulated with the fifth confidence score to obtain the vibration event corresponding to the vibration signal.

[0164] Compared to radar and sound signals, which require complex models for pattern recognition, touch and vibration signals are simple in data type and have clear triggering conditions. Therefore, atomic events can be constructed using threshold judgments and duration judgments. For example, a door / window contact sensor, typically based on a reed switch, can output a binary signal (e.g., 0 or 1, representing open / closed). The edge computing layer's interface module can monitor the signal's level transitions in real time. For a vibration sensor, whose output is an analog voltage signal proportional to vibration intensity, the processing flow can involve setting one or more vibration intensity thresholds. When the voltage signal transmitted from the sensor to the edge computing layer reaches a preset threshold within a certain time window (e.g., duration determines whether it's a slight touch or a violent impact), it can be considered a valid vibration event. Thus, for these simple signals, judgments can be made by setting corresponding rules and thresholds, reducing computational overhead and improving response speed. Furthermore, once a state change is detected or the threshold is exceeded, the system can immediately generate an atomic event with extremely high confidence (e.g., 1).

[0165] Furthermore, for data processing of smart home device status APIs, the smart home system can also obtain the status of other smart home devices in the home environment through local network API interfaces. This acquired data serves as crucial for supplementing contextual information and constructing a complete causal chain. Specifically, the edge computing server acts as a client, communicating with smart devices in the home (such as smart sockets, smart dishwashers, and smart light bulbs) via local area network protocols (such as HTTP API, MQTT, Matter, etc.). This involves periodically polling device status or subscribing to real-time status updates. Since different brands and types of devices return data in different formats via APIs, the smart home system can incorporate an adapter module for the mainstream smart home ecosystem. This module parses and translates device-specific raw data (such as {"power":"on","program":"eco"}) into a unified, standardized atomic event format within the system.

[0166] For example, in conjunction with a "kitchen leak" event, before the user discovers the leak, the tracing engine in the smart home system can find the "start running" event reported by the dishwasher API from historical events, such as:

[0167] {

[0168] "eventId":"evt_api_012",

[0169] "timestamp":"2024-10-27T09:10:05.000Z",

[0170] "source":"API_Dishwasher_Kitchen",

[0171] "eventType":"DeviceStateChange",

[0172] "data":{

[0173] "deviceName":"Kitchen_Dishwasher",

[0174] "state":"running",

[0175] "previous_state":"idle",

[0176] "cycle":"standard_wash",

[0177] "confidence": 1.0 / / API data is considered completely trustworthy

[0178] }}

[0179] The "dishwasher start" event provided by the API with a confidence level of 1.0 can provide a clear "precursor" for the "abnormal pumping sound" and "water flow sound" detected by the subsequent audio module, making the starting point of the causal chain extremely solid, thus enabling the structural causal model to pinpoint the dishwasher as the root cause of the leak with a very high score.

[0180] For the atomic events obtained by the edge computing layer in analyzing the target data, corresponding nodes are created in the causal knowledge graph, and the corresponding atomic events are input into the structural causal model to analyze the causal relationship between two atomic events in order to obtain the causal cohesion score between the two atomic events.

[0181] In some feasible implementations, the event information of atomic events can include trigger time and trigger location. The trigger time can be the timestamp corresponding to the occurrence of the atomic event, and the trigger location can be the location of the atomic event in the home environment (e.g., pre-constructing a map corresponding to the home environment, and then constructing a corresponding coordinate system based on the map to determine the location coordinates of the atomic event). Assuming that the first atomic event is the "cause" event and the second atomic event is the "effect" event, that is, assuming that the occurrence of the first atomic event leads to the occurrence of the second atomic event, the two atomic events and their corresponding event information are input into the structural causal model to calculate the causal cohesion score between the two atomic events, so as to achieve quantitative evaluation and screening. Specifically, the spatiotemporal cohesion between the first atomic event and the second atomic event can be calculated using the first trigger time and first trigger position of the first atomic event and the second trigger time and second trigger position of the second atomic event. Then, the prior causal potential and data-driven correlation between the first atomic event and the second atomic event can be obtained, and the causal cohesion between the first atomic event and the second atomic event can be calculated using the prior causal potential and data-driven correlation. The result confidence of the second atomic event relative to the first atomic event can also be obtained. Then, the causal cohesion score between the first atomic event and the second atomic event in the same candidate causal chain can be calculated using the spatiotemporal cohesion, causal cohesion and result confidence. Thus, the causal relationship between the two atomic events can be effectively analyzed through the causal cohesion score, so as to determine the causal chain corresponding to the occurrence of the family event.

[0182] In the above process, the spatiotemporal cohesion can be calculated by using the first trigger time of the first atomic event and the second trigger time of the second atomic event to obtain the corresponding time difference, and by using the first trigger position of the first atomic event and the second trigger position of the second atomic event to obtain the corresponding spatial distance. Then, the spatiotemporal cohesion between the first atomic event and the second atomic event can be calculated by using the time difference and the spatial distance.

[0183] For dynamic events detected by millimeter-wave radar (such as "human walking" or "pet running"), millimeter-wave radar technology itself possesses precise ranging and angle measurement capabilities. When a radar module detects a dynamic behavior, it not only identifies the type of behavior through the micro-Doppler effect but also calculates the three-dimensional coordinates of the dynamic event's location by analyzing the time of flight and angle of arrival of the echo signal. Therefore, the atomic event output by the radar itself contains highly accurate location information.

[0184] Alternatively, for acoustic events detected by a microphone array (such as "broken glass" or "abnormal noise"), the system uses a "microphone array" rather than a single microphone. By analyzing the time difference of arrival (TDOA) of the same sound at different microphones in the array, the system can use sound source localization algorithms (such as beamforming) to triangulate the direction and approximate location of the sound source. In this way, a sound event is also assigned spatial coordinates.

[0185] For example, for sensors with fixed locations (such as door and window contact sensors, vibration sensors, and smart home appliances): the locations of these devices are pre-configured and known. During the initial installation and deployment of the system, the location information of each fixed sensor (e.g., the coordinates of the "vibration sensor on the living room side table" are (x1, y1, z1), and the coordinates of the "door magnetic sensor at the entrance" are (x2, y2, z2)) are already recorded in the system.

[0186] Therefore, when these sensors trigger events (such as table_vibration), the location of the event directly inherits the fixed coordinates of the sensor in which it occurs, thus determining the trigger location corresponding to the atomic event.

[0187] In practical implementation, for the causal relationships between different atomic events in a causal knowledge graph, the structural causal model can calculate the credibility score between atomic events. For a causal chain, which can be a path composed of a series of different event nodes (i.e., atomic events), its total score can be defined as the product of the scores of all connecting edges on the path, reflecting the strength of the entire evidence chain, such as:

[0188]

[0189] Where any two adjacent events E i (because) and E j The single-step causal cohesion score (STC(E)) between (effects) i →E j It is defined by the following formula:

[0190]

[0191] R(E j ): Result event confidence score (the corresponding confidence score output when an atomic event is identified based on the raw data), representing the "result" event E. j The confidence level of a correctly detected result, with a value of [0, 1], serves as a pre-multiplier to ensure that low-quality, uncertain observations cannot form high-scoring causal chains.

[0192] f r (Δt)·fs (Δd): Spatiotemporal cohesion, which can be used to measure the physical proximity of two events in spacetime. Time decay function f r (Δt)=exp(-λ t ·Δt), where Δt is the time difference, λ t It is the attenuation coefficient; the spatial attenuation function f s (Δd)=exp(-λ s ·Δd), where Δd is the spatial distance, λ s It is the attenuation coefficient.

[0193] f C (E i E j Causal cohesion: This is the core of the model, integrating prior knowledge and data-driven insights, and can be defined as f. C (E i E j )=α·C scm (E i E j )+(1-α)·C gmn (E i E j ), where C scm C represents the prior causal potential based on the structural causal model. gmn Representing the data-driven correlation based on graph neural networks, α is a weighting factor balancing the two. Furthermore, w st and W c : Global weight coefficients. These two coefficients (satisfying w) st +w c =1) Used to adjust the overall importance of "spatial proximity" and "causal logic" in the final score.

[0194] It should be noted that in the above process, the prior causal potential can be information extracted from a pre-set knowledge base. For structural causal models, this can be a pre-defined knowledge base or rule set containing general causal logic. This can be pre-programmed into the smart home system by developers based on physical common sense and / or life experience, containing universally valid causal relationships that are not limited by the home environment. Based on this, when the smart home system evaluates two atomic events (e.g., E...),... i "The object vibrates violently" and E j When determining the relationship between "severe vibration" and "sound of breaking glass", the smart home system can "look up" a table from the knowledge base of the structural causal model. If there is a pre-defined strong causal rule in the structural causal model that points from "violent vibration" to "sound of breaking glass", the system will return a predefined, high potential energy score (e.g., 0.9).

[0195] Correspondingly, data-driven correlation can be calculated by a graph neural network model. This involves analyzing massive amounts of historical event data from the home environment to discover hidden correlation patterns specific to that home environment and not included in the structural causal model. Optionally, the GNN model of a smart home system (such as GraphSAGE) can be continuously trained on accumulated causal knowledge graph data. For example, a GNN might learn from months of data that in this home, whenever a "contact sensor for a specific window is opened," the "indoor temperature sensor reading" is likely to drop within minutes. This can be a home-specific "experience" not found in the structural causal model. When a new "window opened" event occurs, the trained GNN model will predict a strong correlation between it and the "temperature drop" event, outputting a high correlation score (e.g., 0.85).

[0196] Through the above process, the smart home system can calculate the causal cohesion score between two atomic events. The causal cohesion score can effectively measure the causal relationship between two different atomic events. Then, based on the atomic events and their corresponding causal relationships, a causal knowledge graph can be constructed.

[0197] For example, taking "a pet knocking over a vase" as an example, when this series of physical events occurs, the data perception layer and signal processing engine of the smart home system can capture a series of atomic events and create new event nodes in the causal knowledge graph:

[0198] E1:pet_run(time:15:30:01) - detected by millimeter-wave radar;

[0199] E2:table_vibration(time:15:30:02) - detected by a vibration sensor deployed on the side table;

[0200] E3:glass_breaking(time:15:30:03) - detected by microphone array.

[0201] Then, the causal reasoning engine starts working, utilizing STC. 2 The model calculates and establishes the relationship edges between these new nodes:

[0202] ① Spatio-Temporal Edges:

[0203] E1---[precedes]--->E2---[precedes]--->E3 (Time sequence relationship)

[0204] E1---[co_occurs_at]--->SideTable_LivingRoom (Spatial co-occurrence relationship)

[0205] E2---[located_at]--->SideTable_LivingRoom (Spatial Location Relationship)

[0206] ②Causal Edges (likely_causes):

[0207] The system can evaluate and calculate the weights of likely_causes edges.

[0208] Calculation of the edge E1(pet_run)->E2(table_vibration):

[0209] SCM (prior knowledge) might give a lower base score because "running" does not necessarily lead to "vibration".

[0210] The smart home system analyzed historical data over a certain period using a GNN model and discovered a "latent causal pattern": in this household, the probability of the `run` event of the `Pet_01` node and the `vibration` event of the `SideTable_LivingRoom` node occurring simultaneously is much higher than random. The GNN learned this strong correlation specific to this household. Therefore, in STC... 2 In the formula, the data-driven correlation C can be given a very high score by GNN, which significantly improves the final weight of the causal edge E1->E2.

[0211] Calculation of the edge E2(table_vibration)->E3(glass_breaking):

[0212] Prior knowledge (SCM) plays a dominant role here because it contains a strong prior rule: "An object on a certain object (vase)" + "The object vibrates violently," which is highly likely to result in "Fragile items breaking." Therefore, this edge can also achieve a very high STC. 2 Fraction.

[0213] Through the above process, the smart home system can calculate the corresponding atomic events and relational edges, thereby adding nodes and relational edges corresponding to "the pet knocked over the vase" to the causal knowledge graph.

[0214] Step 102: Search for candidate causal chains corresponding to the abnormal event from the causal knowledge graph;

[0215] In this embodiment of the invention, the smart home system can also act as a semantic parser, converting the event query command input by the user into a precise query command for the causal knowledge graph, and then determining the corresponding abnormal event (such as "vase broken") based on the command. Then, it searches for candidate causal chains corresponding to the abnormal event in the causal knowledge graph, so as to select the target causal chain most related to the abnormal event from the candidate causal chains.

[0216] For example, suppose the user inputs the event query command "Why did the vase in the living room break?", the smart home system can determine the abnormal event through semantic analysis as: E_target:[Event C] Vase broken (object: vase_01, location: living room).

[0217] In its implementation, the user inputs relevant information via natural language. The local LLM in the smart home system acts as a semantic parser, converting the user's input into precise retrieval conditions for the causal knowledge graph. These conditions include one or more constraints for locating the E_target within the causal knowledge graph. For example, it might search for an event node where: event type = "broken", associated object = "vase_01", and location = "living room". Then, the smart home system can trace all possible causes starting from the E_target node, using rules such as:

[0218] 1. Tracing direction: Reverse tracing, that is, searching in the opposite direction of the relationship edges such as likely_causes and precededes (finding the cause from the result).

[0219] 2. Path depth: Set a maximum depth (e.g., max_depth = 5 or 10) to constrain the length of the causal chain, prevent searching for too distant or irrelevant causes, and conform to the common sense of "attribution to the nearest cause".

[0220] 3. Edge type filtering: During traversal, only focus on the edge types most relevant to causal reasoning, mainly likely_causes and precededes, while ignoring descriptive relations such as located_at.

[0221] 4. Node type filtering: The nodes in the path should primarily be of the Event type. When traversing to Agent (e.g., person_01) or Object (e.g., table) nodes, the process is usually not continued, but rather they are recorded as participants or objects in the event.

[0222] 5. Spatiotemporal Constraints: Add preliminary spatiotemporal filters. For example, only consider event nodes that occur within one hour before the occurrence of E_target.

[0223] Based on the tracing results, the "path" returned by the graph database is the original network data. The smart home system can also convert it into a more easily processed logical structure. The specific processing procedure can be as follows:

[0224] 1. Path extraction:

[0225] Each path returned by the query is parsed into a sequence consisting of alternating event nodes and relation edges.

[0226] Example path: [Event A: Cat jumps] - (likely_causes) -> [Event B: Impact sound] - (precedes) -> [Event C: Vase breaks]

[0227] 2. Chain assembly and pruning:

[0228] For cases where multiple partially overlapping paths may be found, the smart home system can identify and merge them, forming a "causal tree" or a simplified set of causal chains rooted at E_target. This can be achieved through pruning or by applying simple heuristics for initial filtering, such as:

[0229] Low confidence pruning: If the self-confidence R(E) of an atomic event node in the path is extremely low (e.g., below 0.2), the entire path is discarded directly.

[0230] Spatiotemporal break pruning: If the spatiotemporal distance between consecutive events in the path is abnormally large and exceeds a reasonable threshold, they are discarded.

[0231] 3. Final output:

[0232] Generate a list of candidate causal chains, where each item in the list contains:

[0233] chain_id: The unique identifier of the chain.

[0234] event_sequence: an ordered list of event nodes [E_n, E_{n-1}, ..., E_1, E_target].

[0235] relationship_sequence: The list of corresponding relationship edges [R_n,R_{n-1},...,R_1].

[0236] Each relation edge R_i already has a pre-calculated single-step causal cohesion score STC attached to it. 2 (E_i->E_{i+1}).

[0237] Through the above process, after identifying an abnormal event, the smart home system can utilize the efficient query capabilities of graph databases to quickly "extract" all local causal networks associated with the abnormal event, i.e., candidate causal chains, from the global graph, thereby effectively balancing the comprehensiveness of retrieval and the efficiency of computation.

[0238] Step 103: Based on the causal cohesion score, select the target causal chain from the candidate causal chains for the event query command, and output the event description information corresponding to the target causal chain.

[0239] A causal knowledge graph contains multiple atomic events and the edges between two atomic events (i.e., the causal cohesion scores between two atomic events). For each target atomic event in a candidate causal chain, the target causal cohesion score between two adjacent target atomic events in the same candidate causal chain can be obtained from the causal knowledge graph. The credibility of the candidate causal chain is calculated based on the target causal cohesion score. Finally, the candidate causal chain with the highest credibility is used as the target causal chain for the event query command. By constructing a causal knowledge graph and using the pre-calculated causal cohesion scores for reasoning, efficient and accurate tracing of complex events can be achieved, restoring the corresponding family events for users while ensuring privacy. At the same time, real-time and reliable decision support is achieved, responding to user queries in real time, outputting reliable event descriptions, and further outputting humanized and understandable event description information, greatly improving the usability of information and user experience.

[0240] In some examples, refer to Figure 2 This diagram illustrates the event tracing process provided in this embodiment of the invention. When a physical event occurs, the smart home system collects the corresponding signal through the data perception layer and transmits it to the edge computing layer. The edge computing layer processes the signal to construct a corresponding atomic event stream, which is then input into the corresponding causal inference engine. The causal inference engine calculates the causal cohesion score between two atomic events and constructs a corresponding causal knowledge graph. When a user inputs a query command, the smart home system performs semantic parsing on the query command and queries the corresponding candidate causal chain from the causal knowledge graph based on the parsing results. Then, based on the structural causal model, it calculates the total score corresponding to the candidate causal chain, filters out the target causal chain based on the total score, and finally generates a corresponding report.

[0241] Imagine a user comes home from get off work and finds a broken vase on the side table in the living room, with their pet dog nearby. They can input a query command through the user interaction layer (such as a mobile phone).

[0242] The smart home system identified "broken vase" as the core abnormal event based on the query command, and generated two candidate causal chains through reverse search:

[0243] Chain A (caused by the pet): pet_run->vase_vibration->glass_breaking;

[0244] Chain B (item falls on its own): object_fall(weak signal)->glass_breaking.

[0245] Next, the STC of the two chains can be calculated separately. 2 Total score.

[0246] Chain A achieved a score of 0.87 because all its events were highly similar in time and space, and the GNN provided a moderate correlation between "pet running" and "object vibration". Chain B, on the other hand, had a total score of only 0.15 because its initial event object_fall had a very low confidence level (e.g., 0.3).

[0247] Therefore, the smart home system can identify chain A as the optimal explanation and generate a corresponding time report based on chain A.

[0248] Suppose a user finds water stains of unknown origin on the kitchen floor, but the faucet is turned off. The user enters the corresponding query command through the user interaction layer (such as a mobile phone). The smart home system triggers the tracing engine based on the query command and generates candidate causal chains.

[0249] Chain A (caused by the dishwasher): dishwasher_cycle_start->pump_noise_abnormal->water_detected.

[0250] Chain B (backflow from sink drain pipe): sink_gurgle_sound->water_detected.

[0251] The smart home system analysis revealed that the event sequence in chain A was perfectly sequential in time, and the acoustic characteristics of pump_noise_abnormal and water_detected had a pre-defined strong correlation in SCM (indicating a fault), with its STC... 2 The overall score was 0.92; while the sink_gurgle_sound event in chain B was also detected, its occurrence time was far from that of water_detected, resulting in an extremely low spatiotemporal cohesion score, with a total score of only 0.21. Based on this, the smart home system accurately determined that the problem originated from the dishwasher and generated a corresponding event report.

[0252] It should be noted that the embodiments of the present invention include, but are not limited to, the examples described above. It is understood that, under the guidance of the ideas in the embodiments of the present invention, those skilled in the art can also make settings according to actual needs, and the present invention does not limit such settings.

[0253] In this embodiment of the invention, when a user wants to query the event description corresponding to a family event that has already occurred in the home environment, such as the cause and time of the event, they can input the corresponding event query command. Then, by obtaining the abnormal event corresponding to the event query command and the causal knowledge graph corresponding to the home environment, which includes several atomic events and the causal cohesion scores corresponding to two adjacent atomic events, the system searches for candidate causal chains corresponding to the abnormal event in the causal knowledge graph. Finally, based on the causal cohesion scores, the system selects the target causal chain for the event query command from the candidate causal chains and outputs the event description information corresponding to the target causal chain. Thus, by constructing a causal knowledge graph and using the pre-calculated causal cohesion scores for reasoning, the system can achieve efficient and accurate tracing of complex events, reconstruct the corresponding family events for the user, and ensure privacy. At the same time, it achieves real-time and reliable decision support, can respond to user queries in real time, output reliable event descriptions, and further output humanized and understandable event description information, greatly improving the usability of information and user experience.

[0254] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0255] Reference Figure 3 The diagram shows a structural block diagram of a family event tracing device provided in an embodiment of the present invention, which may specifically include the following modules:

[0256] The response module 301 is used to respond to an event query command for the home environment, obtain the abnormal event corresponding to the event query command, and the causal knowledge graph corresponding to the home environment. The causal knowledge graph includes a number of atomic events and causal cohesion scores corresponding to two adjacent atomic events.

[0257] Search module 302 is used to search for candidate causal chains corresponding to the abnormal event from the causal knowledge graph;

[0258] The feedback module 303 is used to filter out the target causal chain for the event query instruction from the candidate causal chains according to the causal cohesion score, and output the event description information corresponding to the target causal chain.

[0259] In some feasible implementations, the candidate causal chain includes several target atomic events, and the feedback module 303 is specifically used for:

[0260] Obtain the target causal cohesion score between two adjacent target atomic events in the same candidate causal chain, and calculate the credibility of the candidate causal chain based on the target causal cohesion score;

[0261] The candidate causal chain with the highest credibility is selected as the target causal chain for the event query command.

[0262] Among some feasible implementation methods are:

[0263] An event construction module is used to respond to a family event occurring in the family environment, acquire target data associated with the family event, and analyze the target data to obtain the corresponding atomic events;

[0264] The information acquisition module is used to acquire event information corresponding to each of the atomic events;

[0265] The calculation module is used to calculate the causal cohesion score between two atomic events based on the event information;

[0266] The graph construction module is used to create nodes corresponding to the atomic events and use the causal cohesion scores as relation edges to build a causal knowledge graph corresponding to the family environment based on each atomic event.

[0267] In some feasible implementations, the target data includes at least radar signals, the atomic events include at least behavioral events, and the event construction module is specifically used for:

[0268] The radar signal is segmented into frames to obtain the corresponding signal frames;

[0269] Perform a short-time Fourier transform on the signal frame to obtain the corresponding transform result;

[0270] The transformation results corresponding to the consecutive signal frames are arranged in chronological order to obtain the micro-Doppler spectrum of the radar signal.

[0271] The micro-Doppler spectrogram is identified to obtain multiple behavioral labels and a first confidence score;

[0272] The behavior label and the first confidence score are encapsulated to obtain the behavior event corresponding to the radar signal.

[0273] In some feasible implementations, the target data includes at least a sound signal, the atomic event includes at least a sound event, and the event construction module is specifically used for:

[0274] The sound signal is divided into frames to obtain the corresponding audio frames;

[0275] Perform a Fourier transform on the audio frame to obtain the corresponding spectrum;

[0276] The spectrum is converted into the corresponding Mel scale, and the Mel scale corresponding to consecutive audio frames is arranged in chronological order to obtain the corresponding Mel spectrogram;

[0277] Feature extraction is performed on the Mel spectrogram to obtain multiple first sound labels and second confidence scores corresponding to the first sound labels, multiple second sound labels and third confidence scores corresponding to the second sound labels;

[0278] The first sound tag and the second confidence score are encapsulated to obtain the first sound event corresponding to the sound signal, and the second sound tag and the third confidence score are encapsulated to obtain the second sound event corresponding to the sound signal.

[0279] In some feasible implementations, the target data includes at least a touch signal, the atomic event includes at least a touch event, and the event construction module is specifically used for:

[0280] Obtain the touch parameters and the fourth confidence score corresponding to the touch signal;

[0281] Each of the touch parameters is used as a touch tag and encapsulated with the fourth confidence score to obtain the touch event corresponding to the touch signal.

[0282] In some feasible implementations, the target data includes at least a vibration signal, the atomic event includes at least a vibration event, and the event construction module is specifically used for:

[0283] If the duration of the vibration signal within the target time window reaches a preset threshold, then the vibration parameters and the fifth confidence score corresponding to the vibration signal are obtained.

[0284] Each vibration parameter is used as a vibration label and encapsulated with the fifth confidence score to obtain the vibration event corresponding to the vibration signal.

[0285] In some feasible implementations, the event information includes at least the trigger time and trigger location, the atomic events include at least a first atomic event and a second atomic event, and the calculation module is specifically used for:

[0286] The spatiotemporal cohesion between the first atomic event and the second atomic event is calculated by using the first trigger time and the first trigger position of the first atomic event and the second trigger time and the second trigger position of the second atomic event.

[0287] Obtain the prior causal potential and data-driven correlation between the first atomic event and the second atomic event, and use the prior causal potential and the data-driven correlation to calculate the causal cohesion between the first atomic event and the second atomic event.

[0288] Obtain the confidence level of the result of the second atomic event relative to the first atomic event;

[0289] The causal cohesion score between the first atomic event and the second atomic event in the same candidate causal chain is obtained by using the spatiotemporal cohesion, the causal cohesion, and the result confidence score.

[0290] In some feasible implementations, the computing module is specifically used for:

[0291] The time difference is calculated by using the first trigger time of the first atomic event and the second trigger time of the second atomic event;

[0292] The spatial distance is calculated using the first trigger position of the first atomic event and the second trigger position of the second atomic event;

[0293] The spatiotemporal cohesion between the first atomic event and the second atomic event is obtained by calculating the time difference and the spatial distance.

[0294] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0295] In addition, this invention also provides an electronic device, including: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the various processes of the above-described method for tracing the source of family events and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0296] This invention also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the various processes of the aforementioned family event tracing method embodiments and achieves the same technical effects. To avoid repetition, it will not be described again here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0297] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0298] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, EEPROM, Flash, and eMMC, etc.) containing computer-usable program code.

[0299] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0300] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0301] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0302] Although preferred embodiments of the present invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present invention.

[0303] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0304] The above provides a detailed description of a method and apparatus for tracing family events provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for tracing the origins of family events, characterized in that, include: In response to an event query command for a home environment, the system obtains the abnormal event corresponding to the event query command and the causal knowledge graph corresponding to the home environment. The causal knowledge graph includes several atomic events and causal cohesion scores corresponding to two adjacent atomic events. Search the causal knowledge graph for candidate causal chains corresponding to the anomalous event; Based on the causal cohesion score, the target causal chain for the event query instruction is selected from the candidate causal chains, and the event description information corresponding to the target causal chain is output. The method further includes: In response to a family event occurring in the family environment, target data associated with the family event is acquired, and the target data is analyzed to obtain the corresponding atomic events; Obtain the event information corresponding to each of the atomic events; Calculate the causal cohesion score between the two atomic events based on the event information; Create nodes corresponding to the atomic events, and use the causal cohesion scores as relation edges to build a causal knowledge graph corresponding to the family environment based on each atomic event; The event information includes at least the trigger time and trigger location, and the atomic events include at least a first atomic event and a second atomic event. The calculation of the causal cohesion score between the two atomic events based on the event information includes: The spatiotemporal cohesion between the first atomic event and the second atomic event is calculated by using the first trigger time and the first trigger position of the first atomic event and the second trigger time and the second trigger position of the second atomic event. The spatiotemporal cohesion is used to measure the physical spatiotemporal proximity between the two atomic events. The prior causal potential and data-driven correlation between the first atomic event and the second atomic event are obtained, and the causal cohesion between the first atomic event and the second atomic event is calculated using the prior causal potential and the data-driven correlation. The prior causal potential is a score extracted from a preset knowledge base to evaluate the causal relationship between the two atomic events, and the data-driven correlation is calculated by a graph neural network model. Obtain the confidence level of the result of the second atomic event relative to the first atomic event; The causal cohesion score between the first atomic event and the second atomic event in the same candidate causal chain is obtained by using the spatiotemporal cohesion, the causal cohesion, and the result confidence score.

2. The method according to claim 1, characterized in that, The candidate causal chain includes several target atomic events, and the step of filtering the target causal chain for the event query instruction from the candidate causal chains based on the causal cohesion score includes: Obtain the target causal cohesion score between two adjacent target atomic events in the same candidate causal chain, and calculate the credibility of the candidate causal chain based on the target causal cohesion score; The candidate causal chain with the highest credibility is selected as the target causal chain for the event query command.

3. The method according to claim 1, characterized in that, The target data includes at least radar signals, and the atomic events include at least behavioral events. Analyzing the target data to obtain the corresponding atomic events includes: The radar signal is segmented into frames to obtain the corresponding signal frames; Perform a short-time Fourier transform on the signal frame to obtain the corresponding transform result; The transformation results corresponding to the consecutive signal frames are arranged in chronological order to obtain the micro-Doppler spectrum of the radar signal. The micro-Doppler spectrogram is identified to obtain multiple behavioral labels and a first confidence score; The behavior label and the first confidence score are encapsulated to obtain the behavior event corresponding to the radar signal.

4. The method according to claim 1, characterized in that, The target data includes at least an audio signal, and the atomic events include at least audio events. Analyzing the target data to obtain the corresponding atomic events includes: The sound signal is divided into frames to obtain the corresponding audio frames; Perform a Fourier transform on the audio frame to obtain the corresponding spectrum; The spectrum is converted into the corresponding Mel scale, and the Mel scale corresponding to consecutive audio frames is arranged in chronological order to obtain the corresponding Mel spectrogram; Feature extraction is performed on the Mel spectrogram to obtain multiple first sound labels and second confidence scores corresponding to the first sound labels, multiple second sound labels and third confidence scores corresponding to the second sound labels; The first sound tag and the second confidence score are encapsulated to obtain the first sound event corresponding to the sound signal, and the second sound tag and the third confidence score are encapsulated to obtain the second sound event corresponding to the sound signal.

5. The method according to claim 1, characterized in that, The target data includes at least a touch signal, and the atomic event includes at least a touch event. Analyzing the target data to obtain the corresponding atomic events includes: Obtain the touch parameters and the fourth confidence score corresponding to the touch signal; Each of the touch parameters is used as a touch tag and encapsulated with the fourth confidence score to obtain the touch event corresponding to the touch signal.

6. The method according to claim 1, characterized in that, The target data includes at least vibration signals, and the atomic events include at least vibration events. Analyzing the target data to obtain the corresponding atomic events includes: If the duration of the vibration signal within the target time window reaches a preset threshold, then the vibration parameters and the fifth confidence score corresponding to the vibration signal are obtained. Each vibration parameter is used as a vibration label and encapsulated with the fifth confidence score to obtain the vibration event corresponding to the vibration signal.

7. The method according to claim 1, characterized in that, The calculation of the spatiotemporal cohesion between the first atomic event and the second atomic event using the first trigger time and first trigger position of the first atomic event and the second trigger time and second trigger position of the second atomic event includes: The time difference is calculated by using the first trigger time of the first atomic event and the second trigger time of the second atomic event; The spatial distance is calculated using the first trigger position of the first atomic event and the second trigger position of the second atomic event; The spatiotemporal cohesion between the first atomic event and the second atomic event is obtained by calculating the time difference and the spatial distance.

8. A device for tracing the source of family incidents, characterized in that, include: The response module is used to respond to an event query command for the home environment, obtain the abnormal event corresponding to the event query command, and the causal knowledge graph corresponding to the home environment. The causal knowledge graph includes a number of atomic events and causal cohesion scores corresponding to two adjacent atomic events. The search module is used to search for candidate causal chains corresponding to the abnormal event from the causal knowledge graph. The feedback module is used to filter out the target causal chain for the event query instruction from the candidate causal chains based on the causal cohesion score, and output the event description information corresponding to the target causal chain; The device further includes: An event construction module is used to respond to a family event occurring in the family environment, acquire target data associated with the family event, and analyze the target data to obtain the corresponding atomic events; The information acquisition module is used to acquire event information corresponding to each of the atomic events; The calculation module is used to calculate the causal cohesion score between two atomic events based on the event information; The graph construction module is used to create nodes corresponding to the atomic events and use the causal cohesion scores as relation edges to build a causal knowledge graph corresponding to the family environment based on each atomic event. The event information includes at least the trigger time and trigger location, the atomic events include at least a first atomic event and a second atomic event, and the calculation module is specifically used for: The spatiotemporal cohesion between the first atomic event and the second atomic event is calculated by using the first trigger time and the first trigger position of the first atomic event and the second trigger time and the second trigger position of the second atomic event. The spatiotemporal cohesion is used to measure the physical spatiotemporal proximity between the two atomic events. The prior causal potential and data-driven correlation between the first atomic event and the second atomic event are obtained, and the causal cohesion between the first atomic event and the second atomic event is calculated using the prior causal potential and the data-driven correlation. The prior causal potential is a score extracted from a preset knowledge base to evaluate the causal relationship between the two atomic events, and the data-driven correlation is calculated by a graph neural network model. Obtain the confidence level of the result of the second atomic event relative to the first atomic event; The causal cohesion score between the first atomic event and the second atomic event in the same candidate causal chain is obtained by using the spatiotemporal cohesion, the causal cohesion, and the result confidence score.

9. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; The memory is used to store computer programs; When the processor executes a program stored in the memory, it implements the method as described in any one of claims 1-7.

10. A computer-readable storage medium having instructions stored thereon that, when executed by one or more processors, cause the processors to perform the method as described in any one of claims 1-7.

Images