Intelligent doorbell visitor identification method and system based on deep learning

Abstract

The present application discloses a method and system for smart doorbell visitor identification based on deep learning, the method comprising: collecting visitor data, including obtaining visitor images and audio information; processing the visitor data using a self-reflective incremental system that decouples memory and reasoning, and extracting visitor features; performing preliminary identity identification through a reinforcement learning model based on the extracted visitor features; calculating counterfactual Shapley values, and based on the calculation results, explaining and verifying the identification results of the reinforcement learning model; optimizing the identification results using a dual-agent knowledge graph reasoning system in combination with historical data and domain knowledge; and outputting the optimized visitor identity identification results.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and specifically to a method and system for intelligent doorbell visitor identification based on deep learning. Background Art

[0002] With the rapid development of artificial intelligence technology, smart home devices are becoming more and more popular. Among them, smart doorbells, as an important part of home security, have received widespread attention. Traditional smart doorbells mainly rely on simple image recognition technology to identify visitors, which has the following problems:

[0003] Low recognition accuracy: Recognition relies solely on a single image feature and is easily affected by factors such as light and angle, resulting in inaccurate recognition results.

[0004] Lack of learning ability: Unable to self-update and optimize based on historical data and new visitor information, and difficult to adapt to complex and changing actual application scenarios. Summary of the invention

[0005] In view of this, the present application provides a smart doorbell visitor identification method and system based on deep learning, which solves the problems of low accuracy and poor learning ability of smart doorbell visitor identification in the prior art.

[0006] The present application embodiment provides a method for identifying visitors by a smart doorbell based on deep learning, including:

[0007] Collect visitor data, including obtaining visitor image and audio information;

[0008] Processing the visitor data using a self-reflective incremental system with decoupled memory and reasoning to extract visitor features;

[0009] Based on the extracted visitor features, preliminary identity recognition is performed through a reinforcement learning model;

[0010] Calculate the counterfactual Shapley value, and based on the calculation result, explain and verify the recognition result of the reinforcement learning model;

[0011] Utilizing a dual-agent knowledge graph reasoning system, combined with historical data and domain knowledge, to optimize the recognition results;

[0012] Output optimized visitor identification results.

[0013] The steps of collecting visitor data include:

[0014] capturing a facial image of the visitor through a high-definition camera of the smart doorbell;

[0015] Using the microphone array of the smart doorbell to collect the visitor's voice information;

[0016] The visitor data is pre-processed using the edge computing module of the smart doorbell.

[0017] The self-reflective incremental system using decoupled memory and reasoning processes the visitor data and extracts visitor features, including:

[0018] Inputting the visitor data into a deep neural network to extract initial features;

[0019] Inputting the initial features into a memory module, comparing them with historical visitor information, and generating enhanced features;

[0020] Inputting the enhanced features into the reasoning module, combining with current context information, to generate a comprehensive feature vector;

[0021] Using a self-reflection mechanism to evaluate the quality of feature extraction, optimizing the feature extraction process by adjusting the weights of the deep neural network, updating the retrieval strategy of the memory module, and modifying the inference rules of the inference module;

[0022] Based on the optimized feature extraction process, the visitor features are extracted, and the deep neural network parameters are updated through incremental learning, wherein the visitor features are comprehensive feature vectors after the optimized feature extraction process.

[0023] The preliminary identity recognition by using the reinforcement learning model includes:

[0024] Inputting the extracted visitor features into the reinforcement learning model as a state vector;

[0025] Based on the current state, an identification strategy is selected from a predefined action space;

[0026] Execute the selected identity recognition strategy to obtain preliminary recognition results;

[0027] Calculate the reward value based on the accuracy of the recognition result;

[0028] A policy network of the reinforcement learning model is updated based on the reward value.

[0029] The calculating of the counterfactual Shapley value and, based on the calculation result, explaining and verifying the recognition result of the reinforcement learning model include:

[0030] Generate counterfactual scenarios to simulate the impact of different combinations of visitor features on recognition results, wherein the visitor features include facial features, voice features, visit time features, and historical behavior features;

[0031] Calculating the contribution of each of the visitor characteristics to the identification decision;

[0032] Aggregate the contribution of each visitor feature to obtain the importance ranking of visitor features;

[0033] Based on the visitor feature importance, the recognition result of the reinforcement learning model is verified.

[0034] The method of optimizing the recognition result by using the dual-agent knowledge graph reasoning system includes:

[0035] Build visitor information, access patterns, and environmental factors into a knowledge graph;

[0036] Using a guided agent to perform preliminary reasoning in the knowledge graph based on prior knowledge;

[0037] Using an exploration agent to search for potential related information in the knowledge graph;

[0038] Combine the inference results of the guiding agent and the association information of the exploring agent to generate optimized recognition results;

[0039] Update the knowledge graph and add visitor information and association relationships.

[0040] The step of combining the inference result of the guiding agent and the associated information of the exploring agent to generate an optimized recognition result includes:

[0041] Using guided agents to generate preliminary reasoning results based on prior knowledge;

[0042] Use the associated information discovered by the exploration agent to supplement and correct the initial reasoning results;

[0043] Calculate the weight coefficients of the guiding agent's reasoning results and the exploring agent's associated information;

[0044] According to the weight coefficient, the guiding agent reasoning result and the exploring agent association information are integrated;

[0045] Based on the fused information, update the confidence score of the visitor's identity;

[0046] Compare the confidence scores before and after the update, and select the result with the highest confidence as the final optimized recognition result.

[0047] The self-reflective incremental system using decoupled memory and reasoning processes the visitor data, including:

[0048] Use long-term memory storage structure to store historical visitor information;

[0049] Use short-term memory mechanism to temporarily store current interaction information;

[0050] Identifying changes in visitor behavior patterns by comparing the historical visitor information with the current interaction information;

[0051] Update the memory index system based on the identified changes in behavior patterns;

[0052] A multimodal fusion model is used to integrate image and audio information from long-term and short-term memories to generate comprehensive feature representations.

[0053] Inputting the comprehensive feature representation into a decision tree model to perform preliminary visitor identity inference;

[0054] The recognition confidence of the reasoning process was assessed using a metacognitive evaluator;

[0055] When the recognition confidence is below a preset threshold, identifying and classifying the cause of the recognition error;

[0056] Based on the error analysis results, the decision tree model parameters are adjusted using an adaptive learning algorithm;

[0057] The adjusted decision tree model parameters are updated in real time through the online learning framework;

[0058] Using a sample importance evaluation method, high-value new samples are screened from the current interactive information;

[0059] Using the incremental feature extraction method, the feature space is dynamically expanded based on the screened high-value samples to enrich the visitor feature representation.

[0060] The method of optimizing the recognition result by using the dual-agent knowledge graph reasoning system includes:

[0061] Construct an initial knowledge graph containing known visitor identity information, visit frequency, visit time pattern, and relationship with the household owner;

[0062] Receive visitor identification results, and add them to the initial knowledge graph as nodes to be verified;

[0063] Using predefined reasoning rules and historical access patterns, the guiding agent is guided to perform preliminary reasoning in the initial knowledge graph to generate possible identity hypotheses;

[0064] Based on the identity hypothesis generated by the guiding agent, a graph neural network is used to learn representations of the initial knowledge graph to discover potential relationships and patterns;

[0065] Input the discovered potential relationships and patterns into the exploration agent to guide it to conduct in-depth search in the initial knowledge graph;

[0066] Through the attention mechanism, the weights of the guiding agent’s reasoning results and the exploration agent’s search results are dynamically calculated and adjusted;

[0067] Using Monte Carlo tree search, exploring possible reasoning paths in the initial knowledge graph based on the adjusted weights;

[0068] Evaluate and rank the multiple reasoning paths explored;

[0069] Select the inference path with the highest confidence and generate the final visitor identification result;

[0070] Based on the final recognition result, the initial knowledge graph is updated and the weights of relevant nodes and edges are adjusted.

[0071] The embodiment of the present application also provides a smart doorbell visitor identification system based on deep learning, including:

[0072] A collection module is used to collect visitor data, including obtaining visitor images and audio information;

[0073] An extraction module, for processing the visitor data using a self-reflective incremental system that decouples memory and reasoning, and extracting visitor features;

[0074] A preliminary identification module, used for performing preliminary identity recognition through a reinforcement learning model based on the extracted visitor features;

[0075] A verification module, used to calculate the counterfactual Shapley value, and based on the calculation result, explain and verify the recognition result of the reinforcement learning model;

[0076] An optimization module, for optimizing the recognition result by utilizing a dual-agent knowledge graph reasoning system in combination with historical data and domain knowledge;

[0077] The output module is used to output the optimized visitor identification results.

[0078] The present application also provides a computer device, the computer device comprising:

[0079] at least one processor; and,

[0080] a memory communicatively connected to the at least one processor; wherein,

[0081] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above-mentioned method of smart doorbell visitor identification based on deep learning.

[0082] An embodiment of the present application also provides a computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to execute the above-mentioned method of smart doorbell visitor identification based on deep learning.

[0083] An embodiment of the present application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the steps of the above-mentioned method for smart doorbell visitor identification based on deep learning.

[0084] This application has the following technical effects:

[0085] 1. Improved visitor identification accuracy: By comprehensively utilizing image and audio information and combining a self-reflective incremental system that decouples memory and reasoning, the quality of feature extraction and the accuracy of identification are improved.

[0086] 2. Enhanced learning ability of the system: The use of reinforcement learning model and incremental learning method enables the system to continuously learn and optimize from new data, improving the adaptability and generalization ability of the system.

[0087] 3. Improved reasoning capabilities: Using a dual-agent knowledge graph reasoning system, combined with historical data and domain knowledge, deeper and more accurate identity reasoning is achieved.

[0088] 4. Enhanced explainability: By calculating the counterfactual Shapley value, the recognition results of the reinforcement learning model are explained and verified, which improves the transparency and credibility of the system.

[0089] 5. Make full use of knowledge: build and update knowledge graphs, effectively integrate prior knowledge and historical data, and improve the intelligence level of the system.

[0090] 6. Achieved adaptive optimization: Through self-reflection mechanisms and metacognitive evaluation, the system can automatically identify and correct errors and continuously optimize the recognition process. BRIEF DESCRIPTION OF THE DRAWINGS

[0091] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0092] Figure 1 A flowchart of a smart doorbell visitor identification method based on deep learning provided in an embodiment of the present application;

[0093] Figure 2 A module structure diagram of a smart doorbell visitor identification method system based on deep learning provided in an embodiment of the present application;

[0094] Figure 3 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0095] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as they do not conflict with each other.

[0096] Figure 1 is a flow chart of a method for identifying visitor by a smart doorbell based on deep learning in an embodiment of the present invention. Figure 1 As shown, the embodiment of the present invention includes the following steps:

[0097] S1: Collect visitor data, including obtaining visitor image and audio information.

[0098] The purpose of this step is to obtain comprehensive visitor information and provide basic data for subsequent feature extraction and identity recognition. The collection process is as follows:

[0099] S1.1: Capture the visitor’s facial image through the high-definition camera of the smart doorbell.

[0100] Smart doorbells are equipped with high-resolution cameras (such as 1080p or 4K) that can clearly capture visitors' facial images in different lighting conditions (including night vision mode). The camera uses a wide-angle lens (such as a 160-degree viewing angle) to ensure that the entire area of ​​the door can be covered. To cope with visitors of different heights, the camera also has a vertical adjustment function that can automatically adjust the angle to capture the best facial image.

[0101] S1.2: Collect the visitor's voice information using the microphone array of the smart doorbell.

[0102] Smart doorbells are equipped with high-quality microphone arrays, usually containing 2-4 microphones. This design enables sound source localization and noise elimination, improving the quality of voice collection. The microphone array uses beamforming technology to focus on collecting sounds from the direction of the visitor while suppressing ambient noise and echoes. The sample rate of the collected audio signal is 16kHz or higher, and the bit depth is 16 or 24 bits to ensure sufficient audio quality for subsequent voice feature extraction.

[0103] S1.3: Pre-process the visitor data using the edge computing module of the smart doorbell.

[0104] Smart doorbells have built-in edge computing modules, which usually use low-power ARM processors or dedicated AI chips. This module is responsible for preliminary processing of the collected raw data to reduce transmission bandwidth and cloud computing pressure. The preprocessing steps include:

[0105] 1. Image preprocessing:

[0106] Image compression: Use efficient encoding methods such as H.265 to reduce data volume while ensuring quality.

[0107] Face detection: Use a lightweight face detection algorithm (such as an optimized version of MTCNN) to locate the face area.

[0108] Image Enhancement: Adaptive brightness and contrast adjustment based on lighting conditions.

[0109] 2. Audio preprocessing:

[0110] Noise Reduction: Uses an adaptive filtering algorithm to remove background noise.

[0111] Source separation: When there are multiple speakers, a blind source separation algorithm is used to extract the voice of the main speaker.

[0112] Feature extraction: Calculate basic audio features such as Mel-frequency cepstral coefficients (MFCC).

[0113] 3. Data packaging: Pack the pre-processed image and audio data, add timestamps and other metadata (such as doorbell ID, environmental sensor data, etc.).

[0114] 4. Secure encryption: Use encryption algorithms such as AES-256 to encrypt data packets to ensure the security of data transmission.

[0115] Through these preprocessing steps, the smart doorbell can complete some computing tasks locally, reducing the pressure on network transmission and cloud processing, while providing high-quality input data for subsequent feature extraction and recognition tasks.

[0116] S2: Process the visitor data using a self-reflective incremental system with decoupled memory and reasoning to extract visitor features.

[0117] The purpose of this step is to extract high-quality visitor features from the collected multimodal data and continuously optimize the feature extraction process through self-reflection and incremental learning. The core of this system is to decouple the memory and reasoning processes to achieve more flexible and efficient feature extraction.

[0118] A self-reflective incremental system that decouples memory and reasoning is an intelligent system that separates the functions of memory (data storage and retrieval) and reasoning (data processing and decision-making), while continuously evaluating and improving its own performance through a self-reflective mechanism, and is able to incrementally learn new knowledge without forgetting old knowledge.

[0119] Key Features:

[0120] 1. Separation of memory and reasoning: Allows the system to independently optimize data storage and decision-making processes.

[0121] 2. Self-reflection mechanism: The system is able to evaluate its own performance and make adjustments.

[0122] 3. Incremental learning: Ability to continuously learn new knowledge without the need for complete retraining.

[0123] The following sample code will show the basic structure and working principle of this system:

[0124] ```Python

[0125] import numpy as np

[0126] from sklearn.neural_network import MLPClassifier

[0127] from sklearn.metrics import accuracy_score

[0128] class DecoupledMemoryReasoningSystem:

[0129] def __init__(self, input_size, memory_size):

[0130] self.memory = np.zeros((memory_size, input_size))

[0131] self.memory_index = 0

[0132] self.reasoning_model = MLPClassifier(hidden_layer_sizes=(100, 50),max_iter=1000)

[0133] self.performance_history = []

[0134] def store_in_memory(self, data):

[0135] self.memory[self.memory_index] = data

[0136] self.memory_index = (self.memory_index + 1) % len(self.memory)

[0137] def retrieve_from_memory(self, query, k=5):

[0138] distances = np.linalg.norm(self.memory query, axis=1)

[0139] nearest_indices = np.argsort(distances)[:k]

[0140] return self.memory[nearest_indices]

[0141] def reason(self, data):

[0142] relevant_memories = self.retrieve_from_memory(data)

[0143] augmented_data = np.concatenate([data.reshape(1, -1), relevant_memories])

[0144] prediction = self.reasoning_model.predict(augmented_data)

[0145] return prediction[0]

[0146] def learn(self, data, label):

[0147] self.store_in_memory(data)

[0148] relevant_memories = self.retrieve_from_memory(data)

[0149] augmented_data = np.concatenate([data.reshape(1, -1), relevant_memories])

[0150] self.reasoning_model.partial_fit(augmented_data, [label], classes=np.unique(self.reasoning_model.classes_))

[0151] def self_reflect(self, validation_data, validation_labels):

[0152] predictions = [self.reason(data) for data in validation_data]

[0153] performance = accuracy_score(validation_labels, predictions)

[0154] self.performance_history.append(performance)

[0155] if len(self.performance_history) > 1 and performance < self.performance_history[-2]:

[0156] print("Performance decreased. Adjusting memory retrieval...")

[0157] # Here, the adjustment logic for the memory retrieval strategy can be implemented

[0158] def incremental_learn(self, new_data, new_labels):

[0159] for data, label in zip(new_data, new_labels):

[0160] self.learn(data, label)

[0161] self.self_reflect(new_data, new_labels)

[0162] # Usage example

[0163] input_size = 10

[0164] memory_size = 1000

[0165] system = DecoupledMemoryReasoningSystem(input_size, memory_size)

[0166] # Simulate data

[0167] np.random.seed(0)

[0168] train_data = np.random.rand(500, input_size)

[0169] train_labels = np.random.choice([0, 1], 500)

[0170] # Initial training

[0171] system.incremental_learn(train_data, train_labels)

[0172] #Simulate the arrival of new data

[0173] new_data = np.random.rand(100, input_size)

[0174] new_labels = np.random.choice([0, 1], 100)

[0175] # Incremental Learning

[0176] system.incremental_learn(new_data, new_labels)

[0177] # Perform inference

[0178] test_data = np.random.rand(1, input_size)

[0179] prediction = system.reason(test_data[0])

[0180] print(f"Prediction for test data: {prediction}")

[0181] ```

[0182] This sample code implements the basic framework of a self-reflective incremental system that decouples memory and reasoning.

[0183] 1. Memory Module

[0184] The `store_in_memory` method stores new data in memory.

[0185] The `retrieve_from_memory` method retrieves the relevant data from memory using a nearest neighbor search.

[0186] 2. Reasoning Module:

[0187] Use `MLPClassifier` (Multilayer Perceptron Classifier) ​​as the inference model.

[0188] The reason method performs reasoning based on the current input and related data retrieved from memory.

[0189] 3. Learning Process

[0190] The `learn` method stores the new data in memory and updates the inference model.

[0191] Use `partial_fit` for incremental learning and avoid full retraining.

[0192] 4. Self-reflection Mechanism:

[0193] The `self_reflect` method evaluates the performance of the system on the validation data.

[0194] If performance degrades, the system considers adjusting the memory retrieval strategy (this logic has not yet been implemented in the example).

[0195] 5. Incremental Learning:

[0196] The `incremental_learn` method processes new batches of data, updates the system, and triggers self-reflection.

[0197] Advantages of this system include:

[0198] The separation of memory and reasoning allows independent optimization of data storage and retrieval strategies.

[0199] Self-reflection mechanisms enable the system to monitor and improve its own performance.

[0200] Incremental learning capabilities allow the system to continuously adapt to new data without requiring complete retraining.

[0201] In practical applications, this basic framework can be further extended and optimized, for example:

[0202] Implement more sophisticated memory retrieval strategies such as content-based addressing.

[0203] Use more advanced inference models such as deep neural networks or graph neural networks.

[0204] Enhance the self-reflection mechanism so that it can automatically adjust the various components of the system.

[0205] Implement a forgetting mechanism to optimize memory usage and prevent overfitting.

[0206] The following sub-steps are explained in detail:

[0207] S2.1: Input the visitor data into a deep neural network to extract initial features.

[0208] In this step, the embodiment of the present application uses a pre-trained deep neural network model to extract the initial features of the visitor data. For image data, the embodiment of the present application adopts advanced convolutional neural network architectures such as ResNet or EfficientNet. These networks are pre-trained on large-scale face recognition datasets and can effectively capture the key features of the face. For audio data, the embodiment of the present application uses models similar to VGGish or PANN (Pretrained Audio Neural Networks), which can extract rich acoustic features from speech signals. The process of initial feature extraction is parallel, and the image and audio features are processed separately through their respective networks to make full use of computing resources.

[0209] S2.2: Input the initial features into the memory module, compare them with the historical visitor information, and generate enhanced features.

[0210] The memory module uses an architecture similar to that of a Neural Turing Machine, which includes a large external memory and a controller network. Historical visitor information is stored in the external memory in the form of key-value pairs, where the key is the visitor's feature vector and the value contains the visitor's identity information and historical interaction records. When new initial features are input, the controller network calculates the similarity with the stored memory and retrieves the most relevant historical information. Then, the retrieved historical information is fused with the current features through the attention mechanism to generate enhanced features. This process can combine current observations with historical experience to improve the recognition of features.

[0211] For example, the attention mechanism fuses historical information with current features using the following code:

[0212] ```Python

[0213] import torch

[0214] import torch.nn as nn

[0215] import torch.nn.functional as F

[0216] class AttentionFusion(nn.Module):

[0217] def __init__(self, feature_dim, hidden_dim):

[0218] super(AttentionFusion, self).__init__()

[0219] self.query_transform = nn.Linear(feature_dim, hidden_dim)

[0220] self.key_transform = nn.Linear(feature_dim, hidden_dim)

[0221] self.value_transform = nn.Linear(feature_dim, hidden_dim)

[0222] self.attention = nn.MultiheadAttention(hidden_dim, num_heads=4)

[0223] self.output_transform = nn.Linear(hidden_dim, feature_dim)

[0224] def forward(self, current_feature, historical_features):

[0225] # Convert the current feature into a query

[0226] query = self.query_transform(current_feature).unsqueeze(0)

[0227] # Convert historical features into keys and values

[0228] keys = self.key_transform(historical_features).unsqueeze(1)

[0229] values ​​= self.value_transform(historical_features).unsqueeze(1)

[0230] # Apply multi-head attention mechanism

[0231] attn_output, _ = self.attention(query, keys, values)

[0232] # Merge the attention output with the current feature

[0233] fused_feature = self.output_transform(attn_output.squeeze(0)) +current_feature

[0234] return fused_feature

[0235] # Example

[0236] feature_dim = 128

[0237] hidden_dim = 256

[0238] current_feature = torch.randn(feature_dim)

[0239] historical_features = torch.randn(10, feature_dim) # Assume there are 10 historical features

[0240] fusion_model = AttentionFusion(feature_dim, hidden_dim)

[0241] enhanced_feature = fusion_model(current_feature, historical_features)

[0242] print("Enhanced feature shape:", enhanced_feature.shape)

[0243] print("Enhanced feature:", enhanced_feature)

[0244] ```

[0245] In this example:

[0246] 1. This embodiment of the application defines an `AttentionFusion` class, which uses a multi-head attention mechanism to fuse current features and historical features.

[0247] 2. The current features are used as queries, and the historical features are used as keys and values.

[0248] 3. Through the attention mechanism, the system is able to automatically learn which historical features are most relevant to the current features and weight these features accordingly.

[0249] 4. Finally, the embodiment of the present application adds the output of the attention to the original current feature to generate an enhanced feature.

[0250] S2.3: Input the enhanced features into the reasoning module and combine them with the current context information to generate a comprehensive feature vector.

[0251] The reasoning module is implemented based on a graph neural network (GNN), which can effectively process structured and unstructured data. The embodiment of the present application constructs contextual information such as enhanced features, current time, and environmental status (such as weather and light conditions) into a dynamic graph. The nodes of the graph represent different types of information, and the edges represent the relationship between them. Through multi-layer graph convolution operations, the system can automatically learn the complex interactions between features and generate a comprehensive feature vector. This vector not only contains the visual and acoustic features of the visitor, but also integrates contextual information such as time, space, and environment, providing a more comprehensive representation for subsequent identity recognition.

[0252] For example, the multi-layer graph convolution operation in S2.3 learns the complex interactions between features using the following code:

[0253] ```Python

[0254] import torch

[0255] import torch.nn as nn

[0256] import torch.nn.functional as F

[0257] from torch_geometric.nn import GCNConv

[0258] from torch_geometric.data import Data

[0259] class GraphFeatureLearning(nn.Module):

[0260] def __init__(self, input_dim, hidden_dim, output_dim):

[0261] super(GraphFeatureLearning, self).__init__()

[0262] self.conv1 = GCNConv(input_dim, hidden_dim)

[0263] self.conv2 = GCNConv(hidden_dim, hidden_dim)

[0264] self.conv3 = GCNConv(hidden_dim, output_dim)

[0265] def forward(self, data):

[0266] x, edge_index = data.x, data.edge_index

[0267] x = F.relu(self.conv1(x, edge_index))

[0268] x = F.relu(self.conv2(x, edge_index))

[0269] x = self.conv3(x, edge_index)

[0270] return x

[0271] # Usage example

[0272] num_nodes = 10

[0273] input_dim = 16

[0274] hidden_dim = 32

[0275] output_dim = 64

[0276] # Create an example graph

[0277] edge_index = torch.tensor([[0, 1, 1, 2, 2, 3, 3, 4, 4, 5, 5, 6, 6, 7,7, 8, 8, 9, 9, 0],

[0278] [1, 0, 2, 1, 3, 2, 4, 3, 5, 4, 6, 5, 7, 6, 8, 7, 9, 8, 0, 9]], dtype=torch.long)

[0279] x = torch.randn((num_nodes, input_dim))

[0280] data = Data(x=x, edge_index=edge_index)

[0281] model = GraphFeatureLearning(input_dim, hidden_dim, output_dim)

[0282] output = model(data)

[0283] print("Output shape:", output.shape)

[0284] print("Output:", output)

[0285] # Calculate the comprehensive feature vector

[0286] comprehensive_feature = output.mean(dim=0)

[0287] print("Comprehensive feature shape:", comprehensive_feature.shape)

[0288] print("Comprehensive feature:", comprehensive_feature)

[0289] ```

[0290] In this example:

[0291] 1. This embodiment of the application defines a `GraphFeatureLearning` class, which contains three layers of graph convolution operations.

[0292] 2. Each layer of graph convolution learns the interactions between nodes and updates the feature representation of the nodes.

[0293] 3. Through multiple layers of graph convolution, the model can capture high-order interactions between features.

[0294] 4. Finally, the embodiment of the present application generates a comprehensive feature vector by averaging the features of all nodes.

[0295] S2.4: Use a self-reflection mechanism to evaluate the quality of the above feature extraction, and optimize the above feature extraction process by adjusting the weights of the deep neural network, updating the retrieval strategy of the memory module, and modifying the inference rules of the inference module.

[0296] The self-reflection mechanism is a meta-learning process that evaluates the quality of feature extraction through an additional evaluation network. This evaluation network receives the intermediate and final outputs of the feature extraction process and predicts the validity and reliability of the features. Based on the evaluation results, the system automatically adjusts the parameters of each module: for deep neural networks, the network weights are fine-tuned using the gradient descent method; for memory modules, the storage strategy and retrieval algorithm are updated; for reasoning modules, the structure and parameters of the graph neural network are adjusted. This self-reflection mechanism enables the system to continuously optimize its feature extraction process and adapt to different visitors and environmental conditions.

[0297] Specifically include:

[0298] ```Python

[0299] import torch

[0300] import torch.nn as nn

[0301] import torch.optim as optim

[0302] class FeatureExtractor(nn.Module):

[0303] def __init__(self, input_dim, output_dim):

[0304] super(FeatureExtractor, self).__init__()

[0305] self.fc1 = nn.Linear(input_dim, 64)

[0306] self.fc2 = nn.Linear(64, output_dim)

[0307] def forward(self, x):

[0308] x = torch.relu(self.fc1(x))

[0309] x = self.fc2(x)

[0310] return x

[0311] class SelfReflectionMechanism:

[0312] def __init__(self, feature_extractor, learning_rate=0.001):

[0313] self.feature_extractor = feature_extractor

[0314] self.optimizer = optim.Adam(feature_extractor.parameters(), lr=learning_rate)

[0315] self.performance_history = []

[0316] def evaluate_performance(self, features, labels):

[0317] with torch.no_grad():

[0318] outputs = self.feature_extractor(features)

[0319] loss = nn.functional.mse_loss(outputs, labels)

[0320] return loss.item()

[0321] def update_feature_extractor(self, features, labels):

[0322] self.optimizer.zero_grad()

[0323] outputs = self.feature_extractor(features)

[0324] loss = nn.functional.mse_loss(outputs, labels)

[0325] loss.backward()

[0326] self.optimizer.step()

[0327] return loss.item()

[0328] def self_reflect(self, features, labels, validation_features,validation_labels):

[0329] # Evaluate current performance

[0330] current_performance = self.evaluate_performance(validation_features,validation_labels)

[0331] self.performance_history.append(current_performance)

[0332] # If performance drops, try tuning the feature extractor

[0333] if len(self.performance_history) > 1 and current_performance >self.performance_history[-2]:

[0334] print("Performance decreased. Adjusting feature extractor...")

[0335] for _ in range(5): # Try multiple adjustments

[0336] loss = self.update_feature_extractor(features, labels)

[0337] print(f"Adjustment step loss: {loss}")

[0338] # Re-evaluate performance

[0339] new_performance = self.evaluate_performance(validation_features,validation_labels)

[0340] if new_performance < current_performance:

[0341] print("Adjustment successful. Performance improved.")

[0342] else:

[0343] print("Adjustment did not improve performance. Reverting changes.")

[0344] # Rollback logic can be implemented here, such as loading previously saved model parameters

[0345] return current_performance

[0346] # Example

[0347] input_dim = 10

[0348] output_dim = 5

[0349] feature_extractor = FeatureExtractor(input_dim, output_dim)

[0350] self_reflection = SelfReflectionMechanism(feature_extractor)

[0351] # Simulate data

[0352] features = torch.randn(100, input_dim)

[0353] labels = torch.randn(100, output_dim)

[0354] validation_features = torch.randn(20, input_dim)

[0355] validation_labels = torch.randn(20, output_dim)

[0356] # Conduct self-reflection

[0357] for epoch in range(10):

[0358] performance = self_reflection.self_reflect(features, labels,validation_features, validation_labels)

[0359] print(f"Epoch {epoch + 1}, Performance: {performance}")

[0360] ```

[0361] In this example:

[0362] 1. This embodiment of the application defines a simple `FeatureExtractor` as a feature extraction model.

[0363] 2. The `SelfReflectionMechanism` class implements the self-reflection mechanism, including performance evaluation and model adjustment.

[0364] 3. In the `self_reflect` method, the system evaluates the current performance and tries to adjust the feature extractor if it finds that the performance has degraded.

[0365] 4. The tuning process involves updating the model parameters multiple times and then re-evaluating the performance.

[0366] 5. If the performance improves after the adjustment, keep the change; otherwise, you can choose to roll back the change (this part of the logic is not fully implemented in the example).

[0367] S2.5: Based on the optimized feature extraction process, extract the visitor features, and update the deep neural network parameters through incremental learning, wherein the visitor features are comprehensive feature vectors after the optimized feature extraction process.

[0368] After completing self-reflection and optimization, the system will perform the feature extraction process again to generate the final visitor feature. This feature is a high-dimensional vector that contains the visitor's visual, acoustic, and contextual information. At the same time, the system uses an incremental learning strategy to update the parameters of the deep neural network. Specifically, the embodiment of the present application uses an elastic weight consolidation (ElasticWeight Consolidation, EWC) algorithm to maintain memory of old samples while learning new samples. This method can effectively balance the stability and plasticity of the model, enabling the system to continuously learn and adapt to new visitor features without forgetting previously learned knowledge.

[0369] In addition to the above steps, the embodiment of the present application can also be implemented by the following steps A1-A12, which specifically include:

[0370] A1: Use long-term memory storage structure to store historical visitor information.

[0371] The long-term memory storage structure adopts a layered storage architecture, including a fast access layer and a deep storage layer. The fast access layer uses cache technology to store the most recent and most frequently accessed visitor information to improve retrieval efficiency. The deep storage layer uses a distributed storage system that can store a large amount of historical visitor data for a long time. The stored information includes the visitor's feature vector, identity tag, access time pattern, etc. The embodiment of the present application uses a locality-sensitive hashing (LSH) algorithm to build an index to support efficient similarity search. In addition, the system also regularly performs data compression and deduplication operations to optimize the use of storage space.

[0372] A2: Use short-term memory mechanism to temporarily store current interaction information.

[0373] The short-term memory mechanism is implemented based on a recurrent neural network (RNN), in particular using a long short-term memory (LSTM) or a gated recurrent unit (GRU) network. This structure can effectively capture temporal dependencies in sequence data. Current interaction information, such as the visitor's real-time behavior, conversation content, emotional state, etc., is encoded and input into the RNN. The hidden state of the network is continuously updated as the interaction proceeds, maintaining sensitivity to the latest information. At the same time, an embodiment of the present application sets up an attention mechanism that enables the system to dynamically focus on different information based on the current context. The capacity of short-term memory is limited, and old information will gradually be forgotten, ensuring that the system always focuses on the most relevant current information.

[0374] It should be noted that long-term memory and short-term memory can be artificially set with specific time ranges. For example, more than 48 hours is defined as long-term memory, and less than 48 hours is short-term memory.

[0375] A3: By comparing the historical visitor information with the current interaction information, changes in visitor behavior patterns are identified.

[0376] This step uses a dual-stream network structure, one stream processes historical information and the other stream processes current interaction information. The outputs of the two streams are compared and integrated through a fusion module. The fusion module adopts an adaptive weighting mechanism, which can dynamically adjust the importance of different streams according to the relevance and reliability of the information. The embodiment of the present application uses a dynamic time warping (DTW) algorithm to align and compare behavior sequences of different time scales. By calculating the similarity and difference of behavior patterns, the system can identify significant changes in visitor behavior, such as a sudden increase in visit frequency, changes in interaction methods, etc. These changes may imply important events or identity changes, providing important clues for subsequent identity reasoning.

[0377] A4: Update the memory index system based on the identified changes in behavior patterns.

[0378] The memory index system adopts a multi-level index structure, including time index, feature index and relationship index. When a significant change in behavior pattern is detected, the system triggers the index update process. For the time index, the embodiment of the present application uses a tree structure (such as a B+ tree) to efficiently organize and query visitor records of different time scales. The feature index is implemented based on local sensitive hashing (LSH) and supports fast approximate nearest neighbor search. The relationship index uses a graph database to store the relationship network between visitors. The index update process includes inserting new index items, adjusting the weights of existing indexes, and reorganizing the index structure according to the new behavior pattern. The embodiment of the present application adopts an incremental update strategy to only update the affected index part to reduce computing overhead. By updating the index system in a timely manner, it is ensured that the subsequent memory retrieval and reasoning processes can quickly and accurately access the latest behavior pattern information.

[0379] A5: Use a multimodal fusion model to integrate image and audio information from long-term memory and short-term memory to generate a comprehensive feature representation.

[0380] The multimodal fusion model adopts the attention mechanism and the cross-modal transformer architecture. First, the image and audio information are processed by their respective encoder networks to generate initial modality-specific representations. Then, the self-attention mechanism is used to capture the long-range dependencies within each modality. Next, the cross-modal attention layer allows information interaction between different modalities. For example, sound information can guide the system to focus on specific areas in the image, and vice versa. The fusion process also takes into account the temporal difference between long-term memory and short-term memory, using a time-aware attention mechanism to balance the importance of historical information and current information. Finally, a multi-layer perceptron (MLP) is used to map the fused features to a unified feature space to generate a comprehensive feature representation. This representation contains both the long-term characteristics of the visitor and the immediate information of the current interaction.

[0381] Specifically, the goal of the multimodal fusion model is to integrate information from different perceptual modalities (such as vision and hearing) and information from different time scales (long-term memory and short-term memory) to generate a comprehensive feature representation. This process can be analogized to how the human brain integrates multisensory information and past experiences to understand the current situation.

[0382] Take the example of a smart doorbell system identifying visitors to illustrate this process:

[0383] 1. Input information:

[0384] Image information: The currently captured visitor's facial image.

[0385] Audio information: The visitor's voice currently being recorded.

[0386] Short-term memory: Information from recent interactions, such as recent conversation content, behavior patterns, etc.

[0387] Long-term memory: historical visit records, identity information, habitual behaviors, etc.

[0388] 2. Feature extraction:

[0389] Image features: Use convolutional neural networks (CNNs) to extract facial features such as facial contours, expressions, etc.

[0390] Audio features: Use audio processing techniques (such as MFCC) to extract sound features, including timbre, intonation, etc.

[0391] Short-term memory features: Use recurrent neural networks (RNNs) to process recent interaction sequences.

[0392] Long-term memory features: Retrieve relevant historical information from knowledge graphs or databases.

[0393] 3. Multimodal fusion:

[0394] Attention mechanism: Give different weights to information from different modalities. For example, if the current lighting conditions are not good, the system may rely more on audio information.

[0395] Cross-modal learning: learning associations between different modalities. For example, matching the sounds of speech with the movements of the lips.

[0396] Timescale fusion: combining recent information in short-term memory with historical patterns in long-term memory.

[0397] 4. Comprehensive feature generation:

[0398] Finally, the system generates a comprehensive feature vector, which may contain the following information:

[0399] Visitor’s facial feature descriptors;

[0400] The speaker's vocal characteristics;

[0401] recent behavioral patterns (e.g., knocking style, standing posture);

[0402] Historical visit frequency and temporal patterns;

[0403] Inference about the relationship with the householder.

[0404] Let's take a concrete example:

[0405] Suppose a visitor comes in on a Saturday afternoon:

[0406] 1. Image information shows that this is a middle-aged male, dressed casually, holding a toolbox.

[0407] 2. Audio analysis shows that the speaker has a low voice and a friendly tone.

[0408] 3. Short-term memory records show that the visitor had pressed the doorbell 10 minutes ago but received no response.

[0409] 4. Long-term memory data indicate that this visitor appears on the second Saturday afternoon of every month and usually stays for about an hour.

[0410] The multimodal fusion model integrates this information:

[0411] Visual features matched long-term memory, confirming that this was likely a regularly visiting maintenance worker.

[0412] The audio characteristics are consistent with historical records, further strengthening this judgment.

[0413] The repeated bell-ringing behavior in short-term memory matched the regular access pattern in long-term memory.

[0414] The existence of a toolbox is consistent with the identity of a repairman.

[0415] The final comprehensive feature representation may include:

[0416] "There is a high probability (95%) that this person is a monthly regular maintenance worker, with a friendly attitude, performing routine maintenance tasks, and it is recommended that he be allowed to enter."

[0417] This comprehensive feature representation not only includes the visitor's identity information, but also includes behavioral intentions and recommended actions, all of which are derived by fusing multi-modal and multi-temporal information. This rich representation can help the smart doorbell system make smarter and more contextual decisions, such as automatically notifying the homeowner of a maintenance worker's visit or providing specific instructions when the homeowner is not at home.

[0418] A6: Input the comprehensive feature representation into a decision tree model to perform preliminary visitor identity inference.

[0419] The decision tree model uses a random forest or gradient boosting decision tree (such as XGBoost) algorithm. These ensemble learning methods can effectively process high-dimensional features and have good generalization capabilities. In the training phase, the embodiment of the present application uses historical data to construct multiple decision trees, and each tree is trained based on a feature subset and a sample subset. During reasoning, the comprehensive feature representation is input into each decision tree, and each tree makes predictions independently. The final identity reasoning result is obtained by voting or weighted averaging. One advantage of the decision tree model is its interpretability. The embodiment of the present application can analyze the contribution of each feature to the final decision, providing a basis for subsequent verification and optimization. In addition, the model also outputs a confidence score for evaluating the reliability of the reasoning result.

[0420] A7: Use a metacognitive evaluator to evaluate the recognition confidence of the above reasoning process.

[0421] The metacognitive evaluator is an independent neural network model that receives the output of the decision tree model (including the prediction results and the importance scores of each feature) and the intermediate state of the entire reasoning process as input. The evaluator uses a multi-layer perceptron (MLP) structure, and the last layer uses a sigmoid activation function to output a confidence score between 0 and 1. The training of the evaluator adopts a meta-learning method, using the correctness of historical reasoning results as a supervisory signal. During the reasoning process, the evaluator not only considers the output confidence of the decision tree, but also analyzes factors such as the quality of the features and the consistency of the decision path to comprehensively evaluate the reliability of the recognition results. This metacognitive evaluation mechanism can capture the uncertainty that the decision tree model may ignore, and provide important guidance for subsequent error recognition and correction.

[0422] A8: When the recognition confidence is below a preset threshold, the cause of the recognition error is identified and classified.

[0423] When the recognition confidence is lower than the preset threshold (for example, 0.7), the system starts the error analysis process. First, a rule-based expert system is used to preliminarily classify possible causes of errors, such as insufficient feature quality, sparse samples, concept drift, etc. Then, a multi-factor analysis model is used to consider multiple dimensions such as feature importance, decision path, and historical accuracy. This model uses a combination of decision trees and logistic regression to handle complex error patterns. For difficult-to-classify situations, the system enables an anomaly detection module based on a variational autoencoder (VAE) to identify special cases that do not conform to known patterns. Finally, the system generates a detailed error report, including possible error types, weights of influencing factors, and correction suggestions. This report provides specific guidance for subsequent model adjustments.

[0424] A9: Based on the error analysis results, the decision tree model parameters are adjusted using an adaptive learning algorithm.

[0425] The adaptive learning algorithm adopts the framework of online incremental learning, which can dynamically adjust the model according to new data and error analysis results. For the random forest model, the embodiment of the present application uses the Online Random Forest algorithm, which allows a single decision tree to be updated without retraining the entire model. For situations where the importance of features changes significantly, the algorithm will dynamically adjust the feature selection strategy. For the problem of sparse samples, the embodiment of the present application introduces an active learning mechanism to identify samples with large amounts of information for labeling. In order to solve the problem of concept drift, the system uses sliding window technology to gradually eliminate old decision trees and add trees trained based on new data. During the parameter update process, the embodiment of the present application uses elastic weight consolidation technology to retain important historical information while learning new knowledge.

[0426] A10: The adjusted decision tree model parameters are updated in real time through the online learning framework.

[0427] The online learning framework adopts a streaming processing architecture, which can continuously receive new data and update the model in real time. The framework uses an asynchronous update mechanism, which allows the update of model parameters to be carried out in parallel with the reasoning process, minimizing the system response delay. The embodiment of the present application implements a parameter server, which is responsible for managing and synchronizing model parameters in a distributed environment. In order to ensure the stability of the update, a progressive learning rate decay strategy and gradient clipping technology are adopted. In addition, the system also includes a rollback mechanism, which can quickly restore to the previous stable state when a significant performance degradation is detected. Through this real-time update mechanism, the decision tree model can continuously adapt to new visitor patterns and environmental changes, and maintain continuous improvement in recognition performance.

[0428] A11: The sample importance evaluation method is used to screen high-value new samples from the current interaction information.

[0429] The sample importance assessment adopts a strategy based on a combination of uncertainty sampling and diversity sampling. Uncertainty sampling uses information entropy or confidence as an indicator to identify samples that are difficult for the model to determine. Diversity sampling selects highly representative samples in the feature space through the Kernel Density Estimation method. The embodiment of the present application also introduces a novelty detection module, which identifies new samples that are significantly different from existing samples based on the Isolation Forest algorithm. For each sample, the system calculates a comprehensive importance score, taking into account three aspects: uncertainty, diversity, and novelty. Finally, the Top-k selection algorithm is used to screen out the sample set with the highest importance score. This method can effectively reduce data redundancy, focus on the most valuable learning samples, and improve learning efficiency.

[0430] A12: Use the incremental feature extraction method to dynamically expand the feature space and enrich the visitor feature representation based on the screened high-value samples.

[0431] The incremental feature extraction method is based on an autoencoder and a feature generation network. First, the embodiment of the present application uses a variational autoencoder (VAE) to encode high-value samples and learn their potential representations. Then, through contrastive learning techniques, the differences between these new samples and the existing feature space are identified. Based on these differences, the embodiment of the present application designs a feature generation network that is similar to the structure of a generative adversarial network (GAN), but focuses on generating new feature dimensions. The newly generated features are screened through a feature selection module that uses LASSO regularization to ensure that only the most relevant and discriminative features are retained. Finally, the embodiment of the present application uses feature fusion technology to integrate the newly generated features with the original feature space to form an expanded feature representation. This process is progressive, and the system will regularly evaluate the effectiveness of new features and make adjustments as needed to ensure the continuous optimization and expansion of the feature space.

[0432] S3: Based on the extracted visitor features, preliminary identity recognition is performed through a reinforcement learning model.

[0433] The goal of the reinforcement learning model is to make the best decision in the visitor identification process, balancing the recognition accuracy and system resource consumption. The embodiment of the present application adopts the deep Q network (DQN) as the basic architecture, and introduces advanced technologies such as priority experience replay and double Q learning to improve learning efficiency and stability.

[0434] S3.1: Input the extracted visitor features as a state vector into the reinforcement learning model.

[0435] The input state vector is a high-dimensional vector that contains information such as the visitor's visual features, acoustic features, historical behavior patterns, etc. The embodiment of the present application uses an attention mechanism to weight these features so that the model can dynamically focus on the most relevant information. The state vector also includes information about the current system state, such as computing resource utilization, current time, etc., so that the model can consider system constraints when making decisions. In order to process high-dimensional state space, the embodiment of the present application uses an autoencoder for dimensionality reduction to retain the most critical information.

[0436] S3.2: Based on the current state, select an identification strategy from the predefined action space.

[0437] The action space includes different identification strategies, such as using different identification algorithms, adjusting the depth of feature extraction, requesting additional information, etc. The embodiment of the present application uses the ε-greedy strategy to balance exploration and exploitation, gradually reducing the probability of exploration over time. In order to handle large-scale action spaces, the embodiment of the present application adopts action embedding technology to map discrete actions into a continuous embedding space, so that the model can learn the similarities between actions.

[0438] S3.3: Execute the selected identity recognition strategy and obtain preliminary recognition results.

[0439] When executing a strategy, the system will call the corresponding recognition algorithm or adjust system parameters according to the selected action. The recognition process is carried out in stages, and each stage may use a different feature subset or algorithm. The embodiment of the present application implements a dynamic decision tree structure, allowing the model to dynamically select the next recognition strategy based on the intermediate results. The recognition results include the predicted identity label, confidence score, and key node information of the decision process.

[0440] S3.4: Calculate the reward value based on the accuracy of the recognition result.

[0441] The design of the reward function takes into account multiple factors: recognition accuracy (positive rewards for correct recognition and negative rewards for incorrect recognition), recognition confidence (high confidence correct recognition gets higher rewards), resource consumption (low resource consumption strategies get extra rewards), and recognition speed (fast recognition gets time rewards). The embodiment of the present application uses multi-objective optimization techniques to balance these potentially conflicting goals. In addition, a long-term reward mechanism is introduced to consider the impact of current decisions on future recognition tasks and encourage the model to learn the long-term optimal strategy.

[0442] S3.5: Update the policy network of the reinforcement learning model based on the reward value.

[0443] The policy network update adopts the deep deterministic policy gradient (DDPG) algorithm, which is an off-policy algorithm suitable for continuous action space. The embodiment of the present application uses the Prioritized Experience Replay technology to give a higher sampling probability to important experience samples. In order to improve the stability of learning, the target network technology is adopted to regularly soft-update the parameters of the main network to the target network. At the same time, the embodiment of the present application implements an adaptive learning rate adjustment mechanism to dynamically adjust the learning rate according to the performance changes during the training process. In order to prevent over-reliance on a single experience, the embodiment of the present application also introduces data enhancement techniques, such as feature masking and noise injection, to increase the diversity of training samples. Through the comprehensive application of these technologies, the reinforcement learning model can continuously optimize its recognition strategy and adapt to different visitors and environmental conditions.

[0444] S4: Calculate the counterfactual Shapley value, and based on the calculation result, explain and verify the recognition result of the reinforcement learning model.

[0445] Counterfactual Shapley Value is a concept used in the field of causal inference and machine learning interpretability. It is mainly used to evaluate the contribution of a feature to the output of a model, especially under assumptions that are contrary to reality. The calculation of counterfactual Shapley value is usually based on the principle of Shapley value in cooperative game theory, and it is expanded and improved to meet the needs of causal inference and model interpretation.

[0446] The purpose of this step is to provide the explainability of the model decision and verify the reliability of the recognition result. The counterfactual Shapley value method allows the embodiment of the present application to quantify the contribution of each feature to the final decision, so as to understand the decision basis of the model.

[0447] S4.1: Generate counterfactual scenarios to simulate the impact of different combinations of visitor features on the recognition results, where the visitor features include facial features, voice features, visit time features, and historical behavior features.

[0448] The embodiment of the present application uses the Monte Carlo sampling method to generate a large number of counterfactual scenarios. Each scenario represents a variant of a feature combination, created by systematically changing or removing certain features. For facial features, the embodiment of the present application uses a generative adversarial network (GAN) to generate possible variants; for voice features, voice conversion technology is applied; for time and behavior features, a time series generation model is used. These generated scenarios form a rich hypothesis space, allowing the embodiment of the present application to comprehensively evaluate the impact of different features.

[0449] S4.2: Calculate the contribution of each of the visitor characteristics to the identification decision.

[0450] The embodiments of the present application use an approximate Shapley value calculation method, such as the SHAP (SHapley Additive ex Planations) algorithm. This method takes into account the interaction between features and can more accurately reflect the importance of features in complex models. During the calculation process, the embodiments of the present application use the kernel SHAP method, which can achieve a good balance between computational efficiency and accuracy. For high-dimensional features, the embodiments of the present application use feature clustering technology to first calculate the Shapley value of the feature group and then refine it to a single feature to reduce the computational complexity.

[0451] In the smart doorbell system, the purpose of calculating feature contribution is to understand which features are most important to the final visitor identification decision. This not only helps to improve the interpretability of the system, but also helps to optimize the feature extraction and decision-making process. The embodiment of this application uses SHAP (SHapley Additive exPlanations) value to quantify this contribution.

[0452] The SHAP value is based on the concept of Shapley value in game theory. It takes into account the interaction between features and can fairly distribute the contribution of each feature to the prediction results.

[0453] Assume that the smart doorbell system of the embodiment of the present application uses the following features to identify visitors:

[0454] 1. Facial feature similarity (0-1 points);

[0455] 2. Sound matching (0-1 points);

[0456] 3. Regularity of visit time (0-1 points, 1 means completely consistent with historical rules);

[0457] 4. Behavior pattern matching (0-1 points);

[0458] 5. Characteristics of items carried (0-1 points, 1 means completely in line with expectations).

[0459] example:

[0460] At 3 pm one day, a visitor arrived. The features collected by the system are as follows:

[0461] Facial feature similarity: 0.85;

[0462] Sound matching: 0.78;

[0463] Regularity of visit time: 0.92;

[0464] Behavior pattern matching: 0.88;

[0465] Carrying item characteristics: 0.95;

[0466] The final recognition result of the system is: 95% probability that it is a frequent customer "Mr. Zhang" (maintenance worker).

[0467] Now, the present embodiment of the application calculates the contribution of each feature to this 95% probability. After using the SHAP value calculation, the present embodiment of the application obtains the following results:

[0468] 1. Facial feature similarity: SHAP value = 0.25 (26.3%);

[0469] 2. Sound match: SHAP value = 0.15 (15.8%);

[0470] 3. Regularity of access time: SHAP value = 0.22 (23.2%);

[0471] 4. Behavior pattern matching: SHAP value = 0.18 (18.9%);

[0472] 5. Carried items feature: SHAP value = 0.15 (15.8%).

[0473] explain:

[0474] 1. Facial feature similarity (26.3%):

[0475] This is the feature that contributes the most. The high similarity of 0.85 strongly supports the identification of "Mr. Zhang". Its importance reflects the central role of facial recognition in the system.

[0476] 2. Regularity of visit time (23.2%):

[0477] The second most important feature. The high regularity of 0.92 indicates that this visit is very consistent with the historical visit pattern of "Mr. Zhang", which greatly increases the confidence of recognition.

[0478] 3. Behavioral pattern matching (18.9%):

[0479] The third most important feature. The match of 0.88 indicates that the visitor's behavior is very similar to Mr. Zhang's past behavior, further supporting the recognition result.

[0480] 4. Voice matching (15.8%) and characteristics of items carried (15.8%):

[0481] These two features are equally important. Although their raw values ​​are high (0.78 and 0.95 respectively), they have relatively little impact on the final decision. This may be because when identifying "Mr. Zhang", the system relies more on facial features and visit patterns.

[0482] Through this analysis, the following insights can be obtained from the embodiments of the present application:

[0483] 1. Facial features and visit time patterns are the most critical factors in identifying this visitor.

[0484] 2. Behavioral patterns also play an important complementary role.

[0485] 3. Although the sound match is relatively low, it still provides valuable information.

[0486] 4. Although the characteristics of the carried items have a high match degree, their impact on this identification is relatively small, probably because other characteristics have provided sufficient evidence.

[0487] This contribution analysis can help system developers:

[0488] 1. Optimize the feature extraction process and focus on the most influential features.

[0489] 2. Improve the decision model and adjust the weights of different features.

[0490] 3. Provide better explainability so that users can understand how the system makes decisions.

[0491] 4. Identify potential system biases or vulnerabilities, such as if the system is overly dependent on a single feature.

[0492] In this way, the smart doorbell system can continuously optimize its recognition algorithm, improving accuracy and reliability while maintaining a high degree of transparency and explainability.

[0493] S4.3: Aggregate the contribution of each of the visitor features to obtain a visitor feature importance ranking.

[0494] The aggregation process uses a weighted average method, and the weights are based on the stability and consistency of the features. The embodiment of the present application introduces an adaptive weight mechanism that can automatically adjust the feature weights according to different scenarios. In order to deal with the correlation between features, the embodiment of the present application uses principal component analysis (PCA) technology to identify the main independent feature dimensions. The final importance ranking not only considers the contribution of a single feature, but also the synergistic effect of the feature combination. The embodiment of the present application uses visualization techniques such as heat maps and decision tree diagrams to intuitively display the distribution of feature importance.

[0495] S4.4: Based on the importance of the visitor features, verify the recognition results of the reinforcement learning model.

[0496] The verification process includes several key steps: First, the embodiment of the present application compares whether the decision basis of the model is consistent with the expected important features. Secondly, sensitivity analysis is used to check the stability of the model to perturbations of key features. Thirdly, the embodiment of the present application builds a simplified model that uses only the most important features and compares its performance with the complete model. Finally, the embodiment of the present application introduces a human expert review mechanism to allow field experts to evaluate the rationality of the model's decisions. If significant inconsistencies or anomalies are found, the system will mark the recognition result as pending further investigation and may trigger a retraining or fine-tuning process for the model.

[0497] S5: Utilize the dual-agent knowledge graph reasoning system, combine historical data and domain knowledge, and optimize the recognition results.

[0498] The dual-agent knowledge graph reasoning system aims to combine structured knowledge and flexible reasoning to further optimize the recognition results. This system consists of two complementary agents: a guiding agent and an exploring agent, which work together to achieve more accurate and reliable visitor identification.

[0499] S5.1: Build visitor information, access patterns, and environmental factors into a knowledge graph.

[0500] The construction of the knowledge graph adopts a multi-source heterogeneous data integration method. The embodiment of the present application uses ontology learning technology to automatically extract domain concepts and relationships to form the skeleton structure of the graph. Visitor information is represented as entity nodes in the graph, including basic attributes (such as age, gender) and behavioral characteristics. Access patterns are encoded as temporal relationship edges between entities, capturing information such as visit frequency and time regularity. Environmental factors (such as weather and special events) serve as context nodes and establish temporary associations with visitor nodes. The embodiment of the present application uses graph embedding technology (such as TransE or ComplEx) to map entities and relationships in the graph to a low-dimensional vector space to facilitate subsequent similarity calculations and reasoning.

[0501] S5.2: Use a guided agent to perform preliminary reasoning in the knowledge graph based on prior knowledge.

[0502] The guide agent works based on rule reasoning and path sorting algorithms. The embodiment of the present application predefines a series of reasoning rules, such as "people who often visit on weekday mornings may be family members." These rules are converted into path queries on the graph. Using the knowledge graph embedding generation index, the embodiment of the present application applies an efficient path query algorithm (such as a bidirectional search or A* algorithm) to quickly locate entities that meet the rules. At the same time, a confidence scoring mechanism is introduced to assign confidence to each reasoning result based on the reliability and completeness of the path. To handle uncertainty, the embodiment of the present application adopts a probabilistic graph model (such as a Markov logic network) to allow reasoning with uncertainty.

[0503] S5.3: Utilize an exploration agent to search for potential related information in the knowledge graph.

[0504] The exploration agent adopts a method that combines graph neural networks (GNN) and reinforcement learning. The embodiment of the present application uses a graph attention network (GAT) to capture the complex relationships between nodes, allowing the model to dynamically adjust the importance of different relationships. The reinforcement learning part uses the Monte Carlo Tree Search (MCTS) algorithm to perform intelligent exploration in the graph. The reward function design takes into account the novelty of discovering new associations, the relevance to the current recognition task, and the interpretability of the reasoning path. In order to improve the efficiency of exploration, the embodiment of the present application implements a hierarchical search strategy, first exploring at a coarse-grained level, and then performing a fine-grained search in valuable areas. During the exploration process, the knowledge graph is dynamically updated, newly discovered relationships are added, and the graph structure is improved in a rolling manner.

[0505] S5.4: Combine the reasoning results of the guidance agent and the association information of the exploration agent to generate optimized recognition results.

[0506] The combination process adopts an integrated learning framework. The rule-based reasoning results of the guiding agent and the new associations discovered by the exploration agent constitute different "expert opinions". The embodiment of the present application uses the Stacking integration method to train a meta-learner to integrate these opinions. The meta-learner uses a gradient boosting decision tree (such as XGBoost), which is capable of learning complex nonlinear combination rules. During the integration process, the embodiment of the present application considers factors such as the historical accuracy of each agent and the applicability of the current situation, and dynamically adjusts their weights. In addition, an uncertainty quantification module is introduced, which uses a Bayesian neural network to estimate the confidence interval of the integration result to provide a reliability measure for the final decision.

[0507] S5.4 specifically includes:

[0508] S5.4.1: Use a guided agent to generate preliminary inferences based on prior knowledge.

[0509] The process involves using predefined rules and historical data patterns. The boot agent uses a rule-based expert system combined with a fuzzy logic reasoning mechanism. It first retrieves rules from the knowledge base that are relevant to the current situation, such as "If a visitor comes on a weekday morning and carries a toolbox, it is likely to be a maintenance worker." Then, it applies these rules using a forward chaining algorithm while taking uncertainty into account. Fuzzy logic allows for the processing of fuzzy concepts such as "often", "likely", etc. In addition, the boot agent also uses a Bayesian network model to process conditional probability relations and calculate the initial probability distribution of different identity hypotheses. Finally, it generates a list of preliminary reasoning results containing multiple possible identities and their corresponding probabilities.

[0510] S5.4.2: Use the associations discovered by the exploration agent to supplement and correct preliminary inference results.

[0511] The exploration agent uses a graph neural network (GNN) to conduct a deep search in the knowledge graph to discover potential association information. This association information may include the visitor's implicit behavior patterns, social network relationships, or the influence of environmental factors. For example, the exploration agent may find that the visitor often visits under certain weather conditions or has close connections with other known visitors. The supplementation process uses an attention mechanism to compare and integrate these newly discovered association information with the preliminary reasoning results. If significant differences are found, the system triggers a correction process. The correction uses a Bayesian update method to adjust the probability distribution of the preliminary reasoning based on the reliability and relevance of the new information. This process not only corrects potential misidentifications, but also enriches the system's understanding of visitors and provides more comprehensive contextual information.

[0512] S5.4.3: Calculate the weight coefficients of the guiding agent's reasoning results and the exploring agent's associated information.

[0513] The weight calculation uses an adaptive weighting algorithm to dynamically determine the influence of each agent by considering multiple factors. First, the system evaluates the historical accuracy of each agent and uses a sliding window technique to calculate the success rate of the last N decisions. Second, the particularity of the current situation is considered. For example, if it is a visit at an unconventional time, the weight of the exploration agent may increase. Third, using uncertainty estimation, the confidence interval of each agent's output is evaluated through the Monte Carlo dropout method. Finally, a multi-objective optimization algorithm, such as NSGA-II, is applied to balance multiple objectives such as accuracy, novelty, and interpretability to obtain the final weight coefficient. This dynamic weight allocation mechanism enables the system to flexibly respond to different recognition scenarios and make full use of the advantages of each agent.

[0514] S5.4.4: According to the weight coefficient, fuse the guidance agent reasoning results and the exploration agent association information.

[0515] The fusion process uses an integrated learning framework, specifically a combination of soft voting and stacking. The soft voting part uses the calculated weight coefficients to perform a weighted average of the outputs of the two agents. The stacking part trains a meta-learner, usually a gradient boosting decision tree (such as XGBoost or LightGBM), which uses the original outputs of the two agents as features to learn more complex nonlinear combination rules. The fusion process also includes a feature interaction module that uses factorization machines to capture second-order and even higher-order interactions between agent outputs. In order to deal with possible data distribution shifts, the system also implements an online learning mechanism that uses incremental learning algorithms (such as online gradient descent) to continuously fine-tune the fusion model. This multi-level fusion strategy can fully utilize the advantages of both agents to produce more robust and accurate comprehensive results.

[0516] S5.4.5: Update the confidence score of the visitor’s identity based on the fused information.

[0517] The update process uses a Bayesian inference framework to treat the fused information as new evidence to update the prior probability. The system maintains a probability distribution model of visitor identity, using Dirichlet distribution to represent the probability of different identities. Each update calculates the posterior distribution through the conjugate prior to ensure computational efficiency. In order to handle continuous feature spaces, the system also integrates a Gaussian process classifier that can perform smooth probability estimates in high-dimensional feature spaces. In addition, a confidence calibration module is implemented, using Platt scaling or isotonic regression methods to ensure that the output confidence score accurately reflects the actual accuracy. This update process not only takes into account the current recognition results, but also incorporates the accumulated knowledge of historical recognition, resulting in a more reliable and stable confidence assessment.

[0518] S5.4.6: Compare the confidence scores before and after the update, and select the result with the highest confidence as the final optimized recognition result.

[0519] The comparison process uses a hypothesis testing framework, specifically paired t-tests and Wilcoxon signed-rank tests, to evaluate the significant difference between the confidence scores before and after the update. The system also implements a stability analysis module that uses bootstrap to generate multiple subsamples to evaluate the robustness of confidence changes. In order to prevent incorrect decisions caused by overconfidence, an entropy regularization term is introduced to encourage the system to be cautious under high uncertainty. The decision-making process also considers the error cost matrix, weighing decisions based on the actual impact of different types of errors (such as misidentification and missed identification). When making the final choice, the system uses a dynamic programming algorithm to optimize the long-term cumulative benefits, rather than just based on the confidence of a single recognition. If the updated confidence is significantly higher than the threshold (usually set to 95%), the new recognition result is adopted; otherwise, the system may choose to retain the original result or trigger additional verification processes, such as requesting user confirmation. This prudent decision-making mechanism ensures that the system can effectively manage risks and uncertainties while improving recognition accuracy.

[0520] S5.5: Update the knowledge graph and add visitor information and association relationships.

[0521] The update of the knowledge graph adopts an incremental learning strategy. New visitor information is represented as entities and relationships to be inserted. The embodiment of the present application uses online update technology of the graph embedding model, such as Online TransE, to adjust the vector representation of entities and relationships in real time. In order to maintain the consistency of the graph, a constraint checking mechanism is implemented to ensure that the newly added information does not violate the existing ontology rules. For potential conflicts, the embodiment of the present application adopts a conflict resolution strategy based on evidence weights to retain more reliable information. At the same time, a version control system is implemented to record the evolution history of the graph and support rollback operations. The embodiment of the present application also regularly compresses the graph, deletes outdated or irrelevant information, and keeps the graph concise and efficient.

[0522] In addition to the above steps, S5 may also include steps B1-B10:

[0523] B1: Build an initial knowledge graph containing known visitor identity information, visit frequency, visit time pattern, and relationship with the household owner.

[0524] The construction of the initial knowledge graph adopts a combination of bottom-up and top-down methods. The bottom-up part uses data mining technology to extract visitor patterns and relationships from historical records. The embodiment of the present application applies frequent pattern mining algorithms (such as FP-growth) to identify common visit time patterns, and uses association rule learning algorithms to discover potential relationships between visitors. The top-down part uses domain expert knowledge to define core concepts and relationship types. The embodiment of the present application uses ontology engineering tools (such as Protégé) to build domain ontology to ensure the standardization and consistency of knowledge representation. The physical storage of the graph uses a graph database (such as Neo4j) to support efficient graph traversal and query operations. To facilitate subsequent updates and maintenance, the embodiment of the present application implements a modular graph construction process, including steps such as data preprocessing, entity recognition, relationship extraction, and knowledge fusion.

[0525] B2: Receive the visitor identification result and add it to the initial knowledge graph as a node to be verified.

[0526] The process of adding nodes to be verified adopts temporary layer technology. The embodiment of the present application creates a temporary layer on top of the main knowledge graph, and the new recognition results are first added to this layer. The transaction mechanism of the graph database is used to ensure the atomicity of the addition operation. A unique identifier and timestamp are assigned to each node to be verified for easy tracking and management. The embodiment of the present application implements a similarity calculation module, which uses methods such as cosine similarity to quickly identify existing entities in the graph that may be related to new nodes. For potential duplicate entities, the system will mark them and decide whether to merge them through a subsequent verification process. At the same time, the embodiment of the present application applies privacy protection mechanisms, such as differential privacy, to ensure that sensitive information is not leaked when adding new nodes.

[0527] B3: Using predefined reasoning rules and historical access patterns, guide the guided agent to perform preliminary reasoning in the initial knowledge graph to generate possible identity hypotheses.

[0528] The reasoning process of the guided agent is based on a multi-level rule reasoning engine. The first layer uses simple if-then rules to quickly process obvious patterns. The second layer uses fuzzy logic reasoning to handle situations involving uncertainty. The third layer uses probabilistic graphical models (such as Bayesian networks) to infer complex conditional probability relationships. The embodiment of the present application uses a decision tree learning algorithm to automatically extract new rules from historical data and enrich the rule base. A strategy combining forward chaining and backward chaining is used in the reasoning process to improve reasoning efficiency. For each generated identity hypothesis, the system calculates a confidence score, taking into account factors such as the applicability of the rule and historical accuracy. The embodiment of the present application also implements an explanation generation module to provide a readable reasoning path description for each hypothesis.

[0529] B4: Based on the identity hypothesis generated by the bootstrapping agent, a graph neural network is used to perform representation learning on the initial knowledge graph to discover potential relationships and patterns.

[0530] The representation learning process adopts graph neural network (GNN) technology, especially the combination of graph attention network (GAT) and graph convolutional network (GCN). The embodiment of the present application first uses the node2vec algorithm to initialize node embedding to capture the local structural information of the graph. Then, multi-layer GCN is applied for information propagation to learn the high-order features of the nodes. The GAT layer is used to dynamically adjust the importance of different neighbor nodes and improve the model's sensitivity to key information. In order to process heterogeneous graphs, the embodiment of the present application implements a relationship-aware attention mechanism to distinguish different types of edges. Model training adopts self-supervised learning methods, such as edge prediction and node classification tasks, to make full use of graph structural information. The embodiment of the present application also introduces a comparative learning strategy to enhance the model's ability to recognize similar patterns. The learned node representation is processed by dimensionality reduction (such as t-SNE) for visual analysis to help discover potential clusters and abnormal patterns.

[0531] B5: The discovered potential relationships and patterns are fed into the exploration agent to guide it to conduct in-depth searches in the initial knowledge graph.

[0532] The exploration agent uses a reinforcement learning framework to model the graph search process as a Markov decision process. The embodiment of the present application uses a deep Q network (DQN) as the basic algorithm, and the state space includes the characteristics of the current node, information about the visited paths, etc. The action space is defined as the possible next moves, including moving along different types of edges or staying at the current node. The reward function design takes into account the novelty of discovering new relationships, the relevance to the current task, and the efficiency of the search path. In order to balance exploration and utilization, the embodiment of the present application implements an epsilon-greedy strategy to dynamically adjust the exploration probability over time. Monte Carlo Tree Search (MCTS) is used in the search process for long-term planning to improve the strategic nature of the search. The embodiment of the present application also introduces knowledge distillation technology to transfer the experience learned from previous searches to new search tasks to improve learning efficiency.

[0533] B6: Through the attention mechanism, the weights of the guidance agent’s reasoning results and the exploration agent’s search results are dynamically calculated and adjusted.

[0534] The implementation of the attention mechanism is based on a multi-head self-attention model. The embodiment of the present application represents the results of the guiding agent and the exploration agent as a series of vectors, each of which contains a feature description and confidence information of the result. Multi-head attention allows the model to evaluate the importance of the results from different angles. The attention calculation takes into account factors such as the confidence, novelty, and relevance of the result to the current context. The embodiment of the present application introduces a context encoder that encodes the current recognition task and environmental information into a query vector to guide the allocation of attention. In order to handle dynamically changing situations, a time decay mechanism is implemented to reduce the weight of old results over time. The calculation of the attention weight uses a softmax function to ensure that the sum of the weights is 1. The embodiment of the present application also implements an adaptive threshold mechanism that automatically adjusts the number of results to be paid attention to according to the entropy of the attention distribution, striking a balance between comprehensiveness and efficiency.

[0535] B7: Using Monte Carlo tree search, possible reasoning paths are explored in the initial knowledge graph based on the adjusted weights.

[0536] The implementation of Monte Carlo Tree Search (MCTS) adopts the UCT (Upper Confidence Bound for Trees) algorithm. Each node of the tree represents a state in the graph, and the edge represents a possible transition. The selection phase uses the UCB1 (Upper Confidence Bounds) formula to balance exploration and exploitation. The expansion phase takes into account the visit frequency and past rewards of the nodes, and gives priority to expanding branches with potential. The simulation phase uses a lightweight heuristic strategy to quickly evaluate the potential value of the path. The backpropagation phase updates the statistical information of all nodes on the path. In order to improve the search efficiency, the embodiment of the present application implements optimization techniques such as progressive expansion and moving average. At the same time, a parallel MCTS algorithm is introduced to accelerate the search process using a multi-core processor. The embodiment of the present application also designs a dynamic adjustment mechanism to dynamically adjust search parameters such as exploration coefficient and simulation depth according to the findings during the search process.

[0537] B8: Evaluate and rank the multiple reasoning paths explored.

[0538] The evaluation process adopts the multi-criteria decision analysis (MCDA) method. The embodiment of the present application defines a series of evaluation indicators, including path length, node importance, edge reliability, path novelty, etc. The weights of these indicators are determined using the analytic hierarchy process (AHP) to reflect their relative importance in the current task. For each reasoning path, the embodiment of the present application calculates a comprehensive score, considering the weighted sum of all indicators. In order to deal with the correlation and nonlinear relationship between indicators, the embodiment of the present application introduces a fuzzy comprehensive evaluation method, using fuzzy sets and membership functions to describe uncertainty. The sorting process uses a fast non-dominated sorting algorithm to consider the Pareto optimal solution of multiple objectives. The embodiment of the present application also implements a sensitivity analysis module to evaluate the robustness of the sorting results to changes in indicator weights. Finally, a visual report is generated to intuitively display the advantages and disadvantages of different paths, facilitating human-computer collaborative decision-making.

[0539] B9: Select the inference path with the highest confidence and generate the final visitor identification result.

[0540] The selection process adopts a multiple verification strategy. First, the embodiment of the present application uses a confidence interval estimation method to calculate the confidence distribution of each reasoning path. The path with the highest upper bound of the confidence interval is selected as a candidate. Then, these candidate paths are cross-validated, and their performance in different scenarios is evaluated using historical data sets. The embodiment of the present application implements an integrated decision model, combines the prediction results of multiple high-confidence paths, and uses a weighted voting method to derive the final identity. The weight is dynamically adjusted based on the historical accuracy and current confidence of the path. In order to enhance robustness, the embodiment of the present application introduces an anomaly detection mechanism to identify and filter out possible abnormal reasoning results. The final identity recognition result not only includes the predicted identity label, but also comes with a detailed confidence report and decision-making basis description. The embodiment of the present application also designs a human-computer interaction interface, allowing human experts to intervene to review and adjust the final results when necessary.

[0541] B10: Based on the final recognition result, the initial knowledge graph is updated and the weights of relevant nodes and edges are adjusted.

[0542] The updating process of the knowledge graph adopts a method combining incremental learning and graph structure optimization. First, the embodiment of the present application uses an online learning algorithm to update the embedded representation of entities and relationships to reflect the latest recognition results. For newly discovered entities or relationships, the embodiment of the present application adopts transfer learning technology to initialize their representations using the knowledge of similar entities. Graph structure optimization includes edge weight adjustment and node importance update. Edge weight adjustment uses attenuation factors and reinforcement factors to enhance or weaken the importance of certain relationships based on the correctness of the recognition results. Node importance update uses a PageRank variant algorithm, considering the connectivity of the node and the recent activation frequency. The embodiment of the present application implements a conflict resolution mechanism to handle the inconsistency between new and old information, and uses a voting system based on evidence weight to decide which information to retain. In order to maintain the sparsity and efficiency of the graph, the embodiment of the present application regularly performs graph compression operations to remove low-weight redundant edges and long-term inactive nodes. Finally, the embodiment of the present application updates the metadata of the graph, records the timestamp, impact range and major changes of this update, and facilitates subsequent version control and rollback operations.

[0543] For example, imagine a smart doorbell system is processing a new visitor identification task. It is Saturday afternoon at 3 o'clock, and a middle-aged man rings the doorbell. The system immediately begins the complex identification process.

[0544] First, the system builds an initial knowledge graph. This graph contains known visitor information, their visit frequency, time patterns, and relationships with the householder. For example, there is a node in the graph representing "Master Zhang", a regular maintenance worker who usually appears on the second Saturday afternoon of each month. The system also records Master Zhang's appearance, voice characteristics, and typical behavior patterns.

[0545] When a new visitor’s data is collected, the system adds this information to the knowledge graph as a node to be verified. This includes the visitor’s image data (showing a middle-aged man wearing glasses, holding a toolbox), audio data (a deep, friendly voice), and the current time information (3 p.m. on Saturday).

[0546] Next, the guidance agent starts to make preliminary inferences in the knowledge graph. It applies predefined rules, such as "If a visitor comes in on a Saturday afternoon and carries a toolbox, he is likely a regular maintenance worker." Based on these rules and historical visit patterns, the guidance agent generates a preliminary hypothesis: there is an 80% chance that this visitor is Mr. Zhang.

[0547] Meanwhile, the discovery agent conducts a deep search in the knowledge graph. It finds some interesting associations: in the last three months, every time Master Zhang visited, a visitor named "Engineer Li" came to check the equipment the day after. This pattern has not been explicitly documented before, but may be helpful for identification.

[0548] The system uses graph neural networks to learn representations of knowledge graphs. It learns complex relationships between visitor nodes, time nodes, and behavior nodes. For example, it may find that there is a strong correlation between the feature "carrying a toolbox" and the identity of "maintenance personnel."

[0549] The exploration agent feeds these newly discovered relationships and patterns into its search strategy. It might pay special attention to nodes and edges related to scheduled maintenance, such as equipment status records, recent maintenance history, etc.

[0550] Through the attention mechanism, the system dynamically adjusts the weights of the guidance agent and the exploration agent's results. Because the current situation (Saturday afternoon, carrying a toolbox) is highly consistent with Master Zhang's typical pattern, the guidance agent's results are given a higher weight. However, the new pattern discovered by the exploration agent (the connection with Mr. Li) is also given a certain importance.

[0551] The system uses Monte Carlo tree search to explore possible reasoning paths in the knowledge graph. It may simulate multiple scenarios, such as "If this is Master Zhang, will Mr. Li come tomorrow?", "If this is not Master Zhang, what other possible explanations are there?", etc.

[0552] After evaluation and sorting, the system selected the inference path with the highest confidence: the visitor is most likely Mr. Zhang, who came to perform monthly routine maintenance. This conclusion is not only based on surface feature matching, but also takes into account time patterns, behavioral characteristics, and newly discovered work associations.

[0553] Finally, the system updates the knowledge graph. It strengthens the association between Master Zhang and the visit time of Saturday afternoon, adds the working relationship edge between Master Zhang and Engineer Li, and slightly adjusts the weights of some features, such as the importance of "carrying a toolbox" in identifying maintenance personnel.

[0554] After the entire process is completed, the system gives the final recognition result: it is 95% certain that the visitor is Mr. Zhang, who is here to perform monthly maintenance. It also provides a prediction: Mr. Li may visit tomorrow afternoon for a follow-up inspection. This result not only accurately identifies the visitor, but also provides valuable predictive information for future interactions.

[0555] This example shows how a complex knowledge graph reasoning system, combined with prior knowledge and dynamically discovered information, can achieve a highly intelligent and adaptable visitor identification process. The system can not only accurately identify the current visitor, but also learn new patterns from them, continuously optimizing its knowledge structure and reasoning ability.

[0556] S6: Output the optimized visitor identification result.

[0557] This is the final step of the entire recognition process, which aims to present the optimized recognition results in a clear and understandable way. The output process includes the following key components:

[0558] 1. Identity tag: The system gives the final visitor identity, such as "Family Member - Father" or "Courier".

[0559] 2. Confidence score: Use a percentage to indicate how confident the system is about the recognition result, for example "Confidence: 92%".

[0560] 3. Key feature summary: Lists the key features that contribute most to the recognition results, such as "Facial feature matching: high", "Access time pattern: weekday morning", etc.

[0561] 4. Decision path visualization: Use simplified decision trees or flowcharts to visually show how the system reaches the final result. This helps users understand the logical process of identification.

[0562] 5. Uncertainty statement: If there are multiple possible identities, the system will list the suboptimal options and their confidence, such as "Suboptimal guess: Neighbor, Confidence: 15%".

[0563] 6. Action suggestions: Based on the recognition results, the system may give some suggestions, such as "suggest opening the door" or "suggest secondary confirmation".

[0564] 7. Historical comparison: A brief description of the visitor's historical identification records, such as "same as last identification result" or "identity has changed since last time".

[0565] 8. Feedback mechanism: Provide a simple interface that allows users to confirm whether the recognition result is correct, or provide corrections. This feedback will be used for continuous learning and improvement of the system.

[0566] 9. Privacy reminder: If the recognition result involves sensitive information, the system will give a privacy protection reminder.

[0567] 10. System status information: briefly describe the current system working status, such as "model last update time", "system operation status", etc.

[0568] The output results adopt a hierarchical information display strategy, first displaying the most important information (such as identity tags and confidence levels), and then allowing users to expand to view more detailed information as needed. The embodiment of the present application uses natural language generation (NLG) technology to convert technical recognition results into easy-to-understand natural language descriptions. For the visualization part, a responsive design is used to ensure that it can be displayed correctly on different devices (such as smartphones and tablets).

[0569] In addition, the system also provides an API interface to allow other smart home devices or security systems to perform subsequent operations based on these recognition results, such as automatically opening doors, sending notifications, etc. All output information is securely encrypted to ensure that only authorized devices and users can access it.

[0570] Through this comprehensive and user-friendly output method, the embodiments of the present application ensure that the recognition results are not only accurate but also easy to understand and use, thereby maximizing the practical value of the smart doorbell system.

[0571] So far, the embodiments of this application have described in detail all the key steps of the deep learning-based smart doorbell visitor identification method. This system makes comprehensive use of advanced technologies such as multimodal data processing, deep learning, knowledge graphs, and reinforcement learning to achieve high-precision, adaptive, and explainable visitor identification functions. Through continuous learning and optimization, the system can continuously improve its performance and provide users with a safe and reliable smart home experience. Embodiment 2

[0572] like Figure 2 As shown, the embodiment of the present application also provides a smart doorbell visitor identification system 20 based on deep learning, including:

[0573] The collection module 21 is used to collect visitor data, including obtaining visitor image and audio information;

[0574] An extraction module 22, for processing the visitor data using a self-reflective incremental system that decouples memory and reasoning, and extracting visitor features;

[0575] A preliminary identification module 23, used for performing preliminary identity recognition through a reinforcement learning model based on the extracted visitor features;

[0576] A verification module 24, used to calculate the counterfactual Shapley value, and based on the calculation result, explain and verify the recognition result of the reinforcement learning model;

[0577] An optimization module 25, for optimizing the recognition result by utilizing a dual-agent knowledge graph reasoning system in combination with historical data and domain knowledge;

[0578] The output module 26 is used to output the optimized visitor identity recognition result. Embodiment 3

[0579] The present application also provides a computer device, such as Figure 3 As shown, the computer device comprises:

[0580] at least one processor; and,

[0581] a memory communicatively connected to the at least one processor; wherein,

[0582] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above-mentioned deep learning-based smart doorbell visitor identification method.

[0583] An embodiment of the present application also provides a computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to execute the above-mentioned deep learning-based smart doorbell visitor identification method.

[0584] An embodiment of the present application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the steps of the above-mentioned deep learning-based smart doorbell visitor identification method.

[0585] Each module in the above-mentioned deep learning-based smart doorbell visitor identification system can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0586] In one embodiment, a computer device is also provided, which is the deep learning-based smart doorbell visitor identification system mentioned in the above method embodiment. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface.

[0587] Among them, the processor of the computer device is used to provide computing and control capabilities, and can be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, etc., but is not limited thereto. The processor may include one or more processors, for example, one or more central processing units (CPUs). When the processor is a CPU, the CPU may be a single-core CPU or a multi-core CPU. The processor may also include one or more dedicated processors, and the dedicated processor may include a GPU, an FPGA, etc., for accelerated processing. The processor is used to call the program code and data in the memory and execute the steps in the above method embodiment. For details, please refer to the description in the method embodiment, which will not be repeated here.

[0588] The memory of the computer device includes, but is not limited to, non-volatile storage media and internal memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium.

[0589] The input / output interface of the computer device is used to exchange information between the processor and external devices.

[0590] The communication interface of the computer device is used to communicate with an external terminal via a network connection.

[0591] When the computer program is executed by a processor, the above method is implemented.

[0592] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the division of each unit / module is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. The mutual coupling, direct coupling, or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of systems or units, which can be electrical, mechanical or other forms.

[0593] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0594] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable system. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a read-only memory (ROM), or a random access memory (RAM), or a magnetic medium, such as a floppy disk, a hard disk, a tape, a disk, or an optical medium, such as a digital versatile disc (DVD), or a semiconductor medium, such as a solid state disk (SSD), etc.

[0595] The system of the embodiments of the present disclosure can execute the method provided by the embodiments of the present disclosure, and the implementation principles are similar. The actions performed by each module in the system of each embodiment of the present disclosure correspond to the steps in the method of each embodiment of the present disclosure. For the detailed functional description of each module of the system, please refer to the description in the corresponding method shown in the previous text, which will not be repeated here.

[0596] The above is only an optional implementation method for some implementation scenarios of the present disclosure. It should be pointed out that for ordinary technicians in this technical field, without departing from the technical concept of the scheme of the present disclosure, other similar implementation methods based on the technical ideas of the present disclosure are also within the protection scope of the embodiments of the present disclosure.

[0597] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A smart doorbell visitor identification method based on deep learning, characterized in that: The following steps are involved: Collect visitor data, including obtaining visitor image and audio information; Processing the visitor data using a self-reflective incremental system with decoupled memory and reasoning to extract visitor features; Based on the extracted visitor features, preliminary identity recognition is performed through a reinforcement learning model; Calculate the counterfactual Shapley value, and based on the calculation result, explain and verify the recognition result of the reinforcement learning model; Utilizing a dual-agent knowledge graph reasoning system, combined with historical data and domain knowledge, to optimize the recognition results; Output optimized visitor identification results.

2. The method according to claim 1, characterized in that: The steps of collecting visitor data include: capturing a facial image of the visitor through a high-definition camera of the smart doorbell; Using the microphone array of the smart doorbell to collect the visitor's voice information; The visitor data is pre-processed using the edge computing module of the smart doorbell.

3. The method according to claim 1, characterized in that The self-reflective incremental system using decoupled memory and reasoning processes the visitor data and extracts visitor features, including: Inputting the visitor data into a deep neural network to extract initial features; Inputting the initial features into a memory module, comparing them with historical visitor information, and generating enhanced features; Inputting the enhanced features into the reasoning module, combining with current context information, to generate a comprehensive feature vector; Using a self-reflection mechanism to evaluate the quality of feature extraction, and optimizing the feature extraction process by adjusting the weights of the deep neural network, updating the retrieval strategy of the memory module, and modifying the inference rules of the inference module; Based on the optimized feature extraction process, the visitor features are extracted, and the deep neural network parameters are updated through incremental learning, wherein the visitor features are comprehensive feature vectors after the optimized feature extraction process.

4. The method according to claim 1, characterized in that: The preliminary identity recognition by using the reinforcement learning model includes: Inputting the extracted visitor features into the reinforcement learning model as a state vector; Based on the current state, an identification strategy is selected from a predefined action space; Execute the selected identity recognition strategy to obtain preliminary recognition results; Calculate the reward value based on the accuracy of the recognition result; A policy network of the reinforcement learning model is updated based on the reward value.

5. The method according to claim 1, characterized in that The calculating of the counterfactual Shapley value and, based on the calculation result, explaining and verifying the recognition result of the reinforcement learning model include: Generate counterfactual scenarios to simulate the impact of different combinations of visitor features on recognition results, wherein the visitor features include facial features, voice features, visit time features, and historical behavior features; Calculating the contribution of each of the visitor characteristics to the identification decision; Aggregate the contribution of each visitor feature to obtain the importance ranking of visitor features; Based on the visitor feature importance, the recognition result of the reinforcement learning model is verified.

6. The method according to claim 1, characterized in that The method of optimizing the recognition result by using the dual-agent knowledge graph reasoning system includes: Build visitor information, access patterns, and environmental factors into a knowledge graph; Using a guided agent to perform preliminary reasoning in the knowledge graph based on prior knowledge; Using an exploration agent to search for potential related information in the knowledge graph; Combine the inference results of the guiding agent and the association information of the exploring agent to generate optimized recognition results; Update the knowledge graph and add visitor information and association relationships.

7. The method according to claim 6, characterized in that The step of combining the inference result of the guiding agent and the associated information of the exploring agent to generate an optimized recognition result includes: Using guided agents to generate preliminary reasoning results based on prior knowledge; Use the associated information discovered by the exploration agent to supplement and correct the initial reasoning results; Calculate the weight coefficients of the guiding agent's reasoning results and the exploring agent's associated information; According to the weight coefficient, the guiding agent reasoning result and the exploring agent association information are integrated; Based on the fused information, update the confidence score of the visitor's identity; Compare the confidence scores before and after the update, and select the result with the highest confidence as the final optimized recognition result.

8. The method according to claim 1, characterized in that The self-reflective incremental system using decoupled memory and reasoning processes the visitor data, including: Use long-term memory storage structure to store historical visitor information; Use short-term memory mechanism to temporarily store current interaction information; Identifying changes in visitor behavior patterns by comparing the historical visitor information with the current interaction information; Update the memory index system based on the identified changes in behavior patterns; A multimodal fusion model is used to integrate image and audio information from long-term and short-term memories to generate comprehensive feature representations. Inputting the comprehensive feature representation into a decision tree model to perform preliminary visitor identity inference; The recognition confidence of the reasoning process was assessed using a metacognitive evaluator; When the recognition confidence is below a preset threshold, identifying and classifying the cause of the recognition error; Based on the error analysis results, the decision tree model parameters are adjusted using an adaptive learning algorithm; The adjusted decision tree model parameters are updated in real time through the online learning framework; Using a sample importance evaluation method, high-value new samples are screened from the current interactive information; Using the incremental feature extraction method, the feature space is dynamically expanded based on the screened high-value samples to enrich the visitor feature representation.

9. The method according to claim 1, characterized in that: The method of optimizing the recognition result by using the dual-agent knowledge graph reasoning system includes: Construct an initial knowledge graph containing known visitor identity information, visit frequency, visit time pattern, and relationship with the household owner; Receive visitor identification results, and add them to the initial knowledge graph as nodes to be verified; Using predefined reasoning rules and historical access patterns, the guiding agent is guided to perform preliminary reasoning in the initial knowledge graph to generate possible identity hypotheses; Based on the identity hypothesis generated by the guiding agent, a graph neural network is used to learn representations of the initial knowledge graph to discover potential relationships and patterns; Input the discovered potential relationships and patterns into the exploration agent to guide it to conduct in-depth search in the initial knowledge graph; Through the attention mechanism, the weights of the guiding agent’s reasoning results and the exploration agent’s search results are dynamically calculated and adjusted; Using Monte Carlo tree search, exploring possible reasoning paths in the initial knowledge graph based on the adjusted weights; Evaluate and rank the multiple reasoning paths explored; Select the inference path with the highest confidence and generate the final visitor identification result; Based on the final recognition result, the initial knowledge graph is updated and the weights of relevant nodes and edges are adjusted.

10. A smart doorbell visitor identification system based on deep learning, characterized in that: include: A collection module is used to collect visitor data, including obtaining visitor images and audio information; An extraction module, for processing the visitor data using a self-reflective incremental system that decouples memory and reasoning, and extracting visitor features; A preliminary identification module, used for performing preliminary identity recognition through a reinforcement learning model based on the extracted visitor features; A verification module, used to calculate the counterfactual Shapley value, and based on the calculation result, explain and verify the recognition result of the reinforcement learning model; An optimization module, for optimizing the recognition result by utilizing a dual-agent knowledge graph reasoning system in combination with historical data and domain knowledge; The output module is used to output the optimized visitor identification results.

Images