Smart home detection and management method, device and computer equipment

Abstract

The present invention relates to a smart home detection and management method, device and computer equipment. The method: collect initial image data through a smart home device with a camera module and perform preprocessing to obtain standardized image data and target metadata; perform image enhancement processing and joint loss optimization to obtain a target enhanced image; perform target detection to obtain a target bounding box and category information, and extract deep feature vectors and visual appearance features to obtain a multimodal feature representation, and perform template matching on the detected human target to obtain an initial recognition result; perform multi-target tracking to obtain a target trajectory and predicted position, and build a spatiotemporal behavior model based on the target trajectory and predicted position, use a long short-term memory network for sequence processing to obtain a behavior feature sequence, and perform anomaly detection and rule matching on the behavior feature sequence to obtain an event detection result. The implementation of the present invention improves the accuracy and flexibility of event detection.

Description

Technical Field

[0001] The present invention relates to the field of smart home detection technology, and in particular to a smart home detection and management method, device and computer equipment. Background Art

[0002] Computer vision-based home monitoring and management systems are becoming increasingly popular. These systems collect image data through smart cameras installed in the home to achieve real-time monitoring and analysis of the home environment. However, in practical applications, smart home cameras face many challenges.

[0003] The lighting conditions in the home environment are complex and changeable, ranging from bright daylight to dim night, to local strong light and shadows, which can significantly affect the image quality and reduce the accuracy of target detection and recognition. There are many types of targets in home scenes, the postures of people are changeable, and occlusion and rapid movement often occur, which puts higher requirements on target detection and tracking algorithms. Traditional target detection and tracking methods often perform poorly when dealing with these complex scenes, and it is difficult to accurately identify and track the dynamic behavior of family members. In addition, smart home systems need to accurately identify family members and analyze their behavior patterns while protecting privacy, so as to detect abnormal events in time and respond. Since smart home devices usually have limited computing resources and it is difficult to run complex deep learning models, this requires the system to be able to achieve efficient image processing, target detection, identity recognition and behavior analysis under limited computing resources. How to design a comprehensive and efficient smart home detection and management method under these constraints has become a difficult problem that needs to be solved urgently. Summary of the invention

[0004] The objective of the present invention is to provide a smart home detection and management method, device and computer equipment to improve the accuracy and flexibility of event detection.

[0005] To achieve the above object, the present invention provides a smart home detection and management method, comprising the following steps:

[0006] Initial image data is collected and preprocessed by a smart home device with a camera module to obtain standardized image data and target metadata;

[0007] Based on the standardized image data and the target metadata, a dual conditional generative adversarial network is used to perform image enhancement processing and joint loss optimization to obtain a target enhanced image;

[0008] Performing target detection on the target enhanced image to obtain a target bounding box and category information, and based on the target bounding box, using a single convolutional attention module to extract deep feature vectors and visual appearance features to obtain a multimodal feature representation;

[0009] Based on the multimodal feature representation, template matching is performed on the detected human target to obtain an initial recognition result;

[0010] Performing multi-target tracking on the initial recognition result and the target bounding box to obtain a target trajectory and a predicted position;

[0011] A spatiotemporal behavior model is constructed based on the target trajectory and the predicted position, a long short-term memory network is used for sequence processing to obtain a behavior feature sequence, and anomaly detection and rule matching are performed on the behavior feature sequence to obtain an event detection result.

[0012] The present invention also provides a smart home detection and management device, comprising:

[0013] An acquisition module is used to acquire initial image data through a smart home device with a camera module and perform preprocessing to obtain standardized image data and target metadata;

[0014] A processing module, configured to perform image enhancement processing and joint loss optimization using a dual conditional generative adversarial network based on the standardized image data and the target metadata to obtain a target enhanced image;

[0015] An extraction module is used to perform target detection on the target enhanced image to obtain a target bounding box and category information, and based on the target bounding box, a single convolutional attention module is used to extract deep feature vectors and visual appearance features to obtain a multimodal feature representation;

[0016] A matching module, used for performing template matching on the detected human target based on the multimodal feature representation to obtain an initial recognition result;

[0017] A tracking module, used for performing multi-target tracking on the initial recognition result and the target bounding box to obtain a target trajectory and a predicted position;

[0018] The detection module is used to build a spatiotemporal behavior model based on the target trajectory and the predicted position, use a long short-term memory network to perform sequence processing to obtain a behavior feature sequence, and perform anomaly detection and rule matching on the behavior feature sequence to obtain an event detection result.

[0019] The present invention also provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any one of the above methods when executing the computer program.

[0020] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned methods are implemented.

[0021] In summary, the technical solution provided by the present invention can effectively improve the image quality under different lighting conditions and improve the accuracy of subsequent target detection and recognition by using a dual conditional generative adversarial network for image enhancement processing and combining a joint loss optimization strategy. Using a single convolutional attention module, fast and accurate target detection and feature extraction are achieved, adapting to the limited computing resources of smart home devices. The accuracy and robustness of family member recognition are improved by adopting a multimodal feature representation and dynamic template matching strategy, combined with temporal consistency constraints. Through multiple fading factor strong tracking volumetric Kalman filtering and adaptive interactive multi-model algorithm, stable tracking of multiple targets in complex scenes is achieved, and occlusion and rapid motion problems are effectively handled. Based on the long short-term memory network and attention mechanism, a spatiotemporal behavior model is constructed, and a class of support vector machines is combined for anomaly detection, which can accurately capture and analyze the behavior patterns of family members. By integrating the results of anomaly detection and rule matching, a decision tree algorithm is used for comprehensive judgment, which improves the accuracy and flexibility of event detection, and can timely detect and respond to abnormal situations. The present invention allows optimization and tailoring according to different specifications of smart home devices, and has good scalability and adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 is a schematic diagram of the steps of a smart home detection and management method in one embodiment of the present invention;

[0023] Figure 2 is a structural block diagram of a smart home detection and management device in one embodiment of the present invention;

[0024] Figure 3 It is a schematic block diagram of the structure of a computer device according to an embodiment of the present invention.

[0025] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0027] Reference Figure 1 This embodiment provides a smart home detection and management method, including the following steps:

[0028] S1, collecting initial image data through a smart home device with a camera module and preprocessing it to obtain standardized image data and target metadata;

[0029] Specifically, the initial image data is collected by a smart home device with a camera module, and the initial image data is subjected to bidirectional filtering to effectively remove the noise in the image and obtain the denoised image data. The denoised image data is subjected to perspective transformation matrix analysis to achieve geometric correction of the image, and the geometrically corrected image data is generated by correcting the perspective distortion in the image. The geometrically corrected image data is subjected to edge detection to extract significant edge information in the image and form an edge map; then, a Hough transform is performed based on the edge map to extract the target straight line and curve features in the image. The image is segmented based on the target straight line and curve features to generate multiple candidate regions of interest. In order to more accurately determine these regions of interest, a region growth analysis is performed on the candidate regions of interest to refine the segmentation process and obtain finely segmented regions of interest. The current timestamp and the parameter information of the camera are obtained, and these data are associated with the finely segmented regions of interest to generate initial metadata. The light intensity of the current environment is measured based on the built-in light sensor, and the measurement result is merged with the initial metadata to form the target metadata. The finely segmented region of interest is scaled and pixel-valued to ensure consistency and comparability of the image data in subsequent analysis and processing steps, thereby obtaining standardized image data.

[0030] S2, based on the standardized image data and target metadata, a dual conditional generative adversarial network is used to perform image enhancement processing and joint loss optimization to obtain the target enhanced image;

[0031] Specifically, the brightness of the standardized image data is analyzed to obtain a brightness distribution map of the image. Based on the brightness distribution map and the light intensity information contained in the target metadata, a suitable image enhancement strategy is determined and a corresponding enhancement parameter set is generated. The standardized image data is input into the generator of the conditional generative adversarial network in the first stage. The enhancement parameter set is used to adjust the weights of the convolutional layer inside the generator to optimize the image enhancement effect, thereby obtaining an initial enhanced image. The initial enhanced image is channel-joined with the original standardized image data to generate multi-channel input data, and the multi-channel input data is input into the generator of the conditional generative adversarial network in the second stage. Through this step, a fine enhanced image is generated, which has higher detail expression and enhancement effect. In order to improve the visual effect of the image, the fine enhanced image is subjected to global contrast adaptive histogram equalization processing to obtain a contrast optimized image. Based on the contrast optimized image, the structural similarity index (SSIM) and peak signal-to-noise ratio (PSNR) are calculated through the contrast optimized image. These image quality evaluation indicators are used to evaluate whether the quality of the image meets expectations. The fine enhanced image is input into the discriminator of the conditional generative adversarial network in the second stage to calculate the authenticity score of the image. Based on the authenticity score, the adversarial loss value is calculated to evaluate the deviation of the generated image in authenticity. At the same time, the fine enhanced image is compared with the standardized image data pixel by pixel to calculate the pixel loss value. Through high-level feature extraction technology, the perceptual loss value between the fine enhanced image and the standardized image is calculated to reflect the difference in the image at the perceptual level. The adversarial loss value, pixel loss value and perceptual loss value are weighted and summed based on the preset weights to obtain the joint loss function value. The joint loss function value is used for back propagation to adjust and update the parameters of the first and second stage conditional generative adversarial networks so that the generated image is more in line with the expected enhancement effect. After the parameter optimization is completed, the standardized image data is re-processed in two stages based on the optimized network parameters. After the enhancement processing of the first and second stage conditional generative adversarial networks, the target enhanced image is finally obtained.

[0032] S3, performing target detection on the target enhanced image to obtain the target bounding box and category information, and based on the target bounding box, using a single convolutional attention module to extract deep feature vectors and visual appearance features to obtain multimodal feature representation;

[0033] Specifically, by performing a multi-scale pyramid transformation on the target enhanced image, multiple feature maps of different scales are generated to capture information of different levels and scales in the image. Target detection is performed on the feature maps of multiple scales to obtain a preliminary target bounding box and the category probability distribution corresponding to each bounding box. Based on the initial bounding box and category probability distribution, a non-maximum suppression operation is performed to remove overlapping candidate boxes and retain the best bounding box, thereby effectively filtering out redundant detection results and obtaining the target bounding box and category information. The target bounding box is coordinate normalized to obtain a standardized target bounding box. The standardized target bounding box is mapped to the target enhanced image, and the image block of the corresponding area is extracted from the image. The target image block is input into a single convolution attention module, and the attention mechanism is used to weight the important areas in the image to obtain an attention weight map. Based on the attention weight map, the target image block is weighted to obtain an attention enhanced target image block. In order to extract the deep information of the image, multi-layer convolution feature extraction is performed on the attention enhanced target image block to generate a deep feature vector. At the same time, a series of visual feature extraction operations are performed on the target image block with enhanced attention, including the calculation of color histogram, HOG features, and LBP texture features. These visual appearance features characterize the color, shape, and texture information of the target image block from different angles, and can supplement the detailed description in the deep feature vector. The deep feature vector extracted by the convolutional network is spliced ​​with the above-extracted visual appearance features to generate the final multimodal feature representation.

[0034] S4, based on the multimodal feature representation, template matching is performed on the detected human target to obtain the initial recognition result;

[0035] Specifically, the multimodal feature representation is normalized, and the feature vector is standardized to ensure that the features are on the same scale so that they can be effectively compared in the subsequent matching process. Based on the standardized feature vector, a local sensitive hash index is constructed. This structure can greatly speed up the search speed of similar features, thereby constructing a fast retrieval structure. Multiple template features are extracted from the pre-built family member feature library, and these template features represent different feature information of family members. The template features are also normalized to obtain a standardized template feature set. Based on the standardized template feature set, a nearest neighbor search is performed in the fast retrieval structure to find the template feature closest to the current target person feature and generate a candidate matching pair. The cosine similarity is calculated for the candidate matching pairs. Cosine similarity can measure the similarity between two feature vectors to obtain a similarity score. In order to improve the accuracy of recognition, an adaptive threshold is set based on the similarity score, and matching pairs below the threshold are filtered out to obtain preliminary matching results. For the retained initial matching results, geometric consistency verification is performed. Geometric consistency verification is used to ensure that the matching pairs are also reasonable in spatial layout, further reducing the possibility of mismatching. After the geometric verification is passed, the weighted voting score is calculated based on the verification results to generate an identity candidate list. The list contains multiple identity candidates that best match the target person. For each candidate identity in the identity candidate list, the corresponding historical recognition records are extracted and a temporal consistency constraint is constructed. By analyzing these historical records, a temporal weighted score is calculated for each candidate identity to evaluate whether the identity meets the recognition pattern of the target person in the past recognition. The temporal weighted score is fused with the matching score of the current frame to obtain a comprehensive score. Finally, all candidate identities are sorted based on the comprehensive score, and the identity with the highest score is selected as the final initial recognition result.

[0036] S5, performing multi-target tracking on the initial recognition results and the target bounding box to obtain the target trajectory and predicted position;

[0037] Specifically, the initial recognition result and the target bounding box are data associated, and the Hungarian algorithm is used to match the target in the current frame with the previous historical trajectory to generate a matching pair list. Based on the matching pair list, multiple volumetric Kalman filters are initialized, each filter corresponds to a tracking target, and an initial filter set is formed. In the target tracking process, each filter is tracked with multiple fading factors. The fading factor is introduced in the time update and measurement update stages, which helps to enhance the robustness of tracking and adapt to the motion changes of the target in different scenes. Through this processing, the corrected state estimate and error covariance are obtained. Based on the corrected parameters, the new information sequence is calculated to measure the matching index of the model. The Markov transfer matrix in the interactive multi-model algorithm is dynamically adjusted according to the model matching index to obtain the adaptive transition probability. The transition probability reflects the transformation relationship between different motion models, and based on the transition probability, multiple motion models are probability-weighted fused to generate a mixed state estimate and a mixed error covariance. The mixed state estimate represents the comprehensive prediction result after the fusion of multiple models, which can more accurately reflect the motion state of the target. The mixed state estimate is subjected to singular value decomposition to obtain a stable state estimation result. Based on the stable state estimation results, the position and speed of the target in the next frame are predicted to form the predicted state of the target. The predicted state of the target is compared with the actual observed value, the prediction error is calculated, and the anomaly detection index is obtained. Based on the anomaly detection index, it is determined whether the target has abnormal conditions such as occlusion or rapid movement. If an abnormality is detected, the system will trigger the target re-identification module to update the target's identity information to ensure the consistency and accuracy of tracking, especially when the target disappears or reappears. In order to improve the stability of the target trajectory, the stable state estimation results of multiple consecutive frames are smoothed to generate a smooth trajectory of the target. Based on the smoothed trajectory, the motion features of the target are extracted, including information such as the target's speed, acceleration, and direction of movement, to obtain the target's motion pattern. The target's motion pattern is associated with the updated target identity information to generate an accurate target trajectory and predicted position.

[0038] S6, builds a spatiotemporal behavior model based on the target trajectory and predicted position, uses a long short-term memory network for sequence processing to obtain a behavior feature sequence, and performs anomaly detection and rule matching on the behavior feature sequence to obtain event detection results.

[0039] Specifically, the target trajectory is sampled in time and space to obtain a discretized trajectory sequence. Through the discretization process, the continuous target trajectory data is effectively simplified into discrete time and space points, and the motion features between adjacent points, such as speed, acceleration and moving direction, are calculated based on the discrete time and space points to generate a motion feature sequence. The motion feature sequence is combined with the predicted position of the target to form an initial time and space behavior feature. The initial time and space behavior feature is processed by sliding window to divide the feature sequence into multiple time segments. Each time segment represents the behavior feature within a certain time range. The features within the time segment are statistically analyzed to extract statistical features such as mean and variance to obtain a statistical feature vector. In order to enhance the feature representation, the statistical feature vector is concatenated with the initial time and space behavior feature to generate an enhanced time and space behavior feature. The enhanced time and space behavior feature is input into the pre-trained long short-term memory network. The long short-term memory network can effectively capture the temporal dependency in the sequence data. Through the processing of the behavior feature by the model, the hidden state sequence of the target behavior is obtained. The hidden state sequence contains the behavior information of the target in different time periods. The attention mechanism is used to weight the hidden state sequence to generate an attention-weighted hidden state sequence. The weighted hidden state sequence is mapped to the feature space through the fully connected layer to obtain the behavior feature sequence. The behavior feature sequence is input into the support vector machine model, and the anomaly score is generated by calculating the distance from the behavior feature sample to the decision boundary. This score represents the degree of abnormality of the current behavior. If the anomaly score exceeds the preset threshold, the behavior is marked as abnormal behavior, and the initial anomaly detection result is generated. In order to analyze these detection results, the predefined event patterns and corresponding trigger conditions are obtained from the rule library. Based on these rules, the behavior feature sequence is matched to check whether the specific behavior conforms to the predefined event pattern. If the behavior feature meets the trigger condition in the rule library, the rule matching result is generated. The rule matching result is fused with the initial anomaly detection result, and the type of event is determined by combining the information of the two. Based on the event type determination result, the corresponding response action is queried from the preset event response strategy library, and the appropriate response instruction is selected. In this way, not only the event type can be determined, but also the corresponding event response instruction can be quickly generated. After combining the instruction with the event type determination result, a complete event detection result is finally generated.

[0040] In one example, initial image data is collected and preprocessed by a smart home device with a camera module to obtain standardized image data and target metadata, including:

[0041] Initial image data is collected by a smart home device with a camera module, and bidirectional filtering is performed on the initial image data to obtain denoised image data;

[0042] Perform perspective transformation matrix analysis on the denoised image data to obtain geometrically corrected image data;

[0043] Perform edge detection on the geometrically corrected image data to obtain an edge map, and perform Hough transform based on the edge map to obtain target straight line and curve features;

[0044] Perform image segmentation based on the target straight line and curve features to obtain multiple candidate regions of interest, and perform region growing analysis on the candidate regions of interest to obtain finely segmented regions of interest;

[0045] Get the current timestamp and camera parameter information, and associate them with the finely segmented region of interest to obtain initial metadata. Then measure the ambient light intensity based on the built-in light sensor, add the measurement results to the initial metadata, and obtain target metadata.

[0046] The finely segmented region of interest is scale normalized and pixel value normalized to obtain standardized image data.

[0047] In this example, the initial image data is collected by a smart home device with a camera module. The collected initial image data is processed by bidirectional filtering. This filtering method combines the information in the spatial domain and the frequency domain, and can effectively remove high-frequency noise in the image while maintaining the image details to obtain denoised image data. Bidirectional filtering considers the similarity of space and pixel intensity. The pixel value after filtering It can be calculated using the following formula:

[0048] ;

[0049] in, is the original value of the adjacent pixels, Represents pixels Neighborhood of is a normalization factor to ensure that the intensity of the filtered image remains within a reasonable range. Control the weight of spatial distance, Controls the weight of pixel intensity differences. Perform perspective transformation matrix analysis on the denoised image data. Perspective distortion is caused by the angle difference between the camera and the scene, which causes the shape and size of objects in the image to appear inconsistent with reality. The perspective transformation matrix can remap the image to correct these distortions. Assume that the point in the original image is , the position after perspective transformation is , then the perspective transformation can be expressed as a 3x3 matrix

[0050] ;

[0051] in is the perspective transformation matrix, which contains all the parameters of perspective distortion. By calibrating some known reference points in the image, the matrix is ​​calculated , correct the distorted image into geometrically accurate image data. The corrected image can more accurately reflect the actual spatial position and shape of the object. Perform edge detection on the geometrically corrected image data in perspective distortion correction. Edge detection is to identify the edges in the image by calculating the rate of change of the image grayscale value. Commonly used algorithms include Sobel, Canny, etc. The edge map can highlight the shape contours in the image and is an important basis for subsequent image feature extraction. After obtaining the edge map, the Hough transform is performed based on the edge map. The Hough transform is an algorithm that parameterizes the straight line and curve features in the image. For example, each straight line in the image can be represented by a pair of parameters in the Hough space. Indicates that Represents the vertical distance between the line segment and the origin of the image coordinate system. Represents the angle between the line segment and the horizontal axis. In this way, the straight line and curve features in the image are extracted. Based on the target straight line and curve features, the image is segmented and divided into several regions of interest. These regions may contain target objects that the smart home system needs to focus on analyzing. The segmented candidate regions of interest are processed by the region growing algorithm. The region growing algorithm starts from an initial seed point and continuously adds similar adjacent pixels to the region until the similarity condition is not met, and the finely segmented regions of interest are obtained. The current timestamp and camera parameter information (such as focal length, viewing angle, etc.) are obtained, and these information are associated with the finely segmented regions of interest to generate initial metadata. At the same time, the light intensity of the current environment is measured based on the built-in light sensor. Changes in lighting conditions may affect the visual effect of the image. The light measurement results are added to the initial metadata to generate target metadata containing lighting information. The finely segmented regions of interest are scaled and pixel values ​​are standardized to obtain standardized image data. Scale normalization is to ensure that target regions of different sizes have consistent scales when processed, thereby eliminating size differences caused by different distances or angles. The normalization process adjusts all pixel values ​​to the same range, usually between 0 and 1, by performing a linear transformation on the pixel values.

[0052] In one example, based on the standardized image data and the target metadata, a dual conditional generative adversarial network is used to perform image enhancement processing and joint loss optimization to obtain a target enhanced image, including:

[0053] Perform brightness analysis on the standardized image data to obtain a brightness distribution map, and determine an image enhancement strategy based on the brightness distribution map and the light intensity information in the target metadata to obtain an enhancement parameter set;

[0054] The standardized image data is input into the generator of the first-stage conditional generative adversarial network, and the convolutional layer weights of the generator are adjusted based on the enhanced parameter set to obtain the initial enhanced image;

[0055] Channel concatenation is performed on the initial enhanced image and the standardized image data to obtain multi-channel input data, and the multi-channel input data is input into the generator of the second-stage conditional generative adversarial network to obtain a refined enhanced image;

[0056] Perform global contrast adaptive histogram equalization on the fine enhanced image to obtain a contrast optimized image, and calculate the structural similarity index and peak signal-to-noise ratio based on the contrast optimized image to obtain the image quality evaluation index;

[0057] Input the refined enhanced image into the discriminator of the second-stage conditional generative adversarial network to obtain a authenticity score, and calculate the adversarial loss based on the authenticity score to obtain the adversarial loss value;

[0058] Calculating pixel loss values ​​for the fine enhanced image and the normalized image data, and extracting perceptual loss values ​​for the fine enhanced image and the normalized image data;

[0059] Based on the preset weights, the adversarial loss value, pixel loss value and perceptual loss value are weighted and summed to obtain the joint loss function value, and the joint loss function value is back-propagated to update the parameters of the first-stage and second-stage conditional generative adversarial networks to obtain the optimized network parameters;

[0060] Based on the optimized network parameters, the standardized image data is reprocessed in two stages to obtain the target enhanced image.

[0061] In this example, brightness analysis is performed on the standardized image data to evaluate the uniformity and intensity variation of the illumination in the image, and a brightness distribution map is generated to help the system identify areas of uneven brightness in the image, especially in environments with complex lighting conditions, such as areas where some image areas are too bright or too dark, resulting in loss of details. Combined with the illumination intensity information in the target metadata, these metadata include the actual illumination measurements of the current environment (such as the light intensity provided by the light sensor). Based on the brightness distribution map and the illumination intensity information, a suitable image enhancement strategy is determined and an enhancement parameter set is generated. The standardized image data is input into the generator of the first stage of the conditional generative adversarial network. The conditional generative adversarial network is a deep learning model that adds conditional information (such as an enhancement parameter set) to the generator network, and can adjust the behavior of the generator according to the input enhancement parameters. In this step, the enhancement parameter set is used to adjust the convolutional layer weights of the generator so that it can generate more optimized images for different brightness conditions. The generator processes the image through multiple convolutional layers, gradually corrects the brightness and contrast of the image, and outputs the initial enhanced image. The initial enhanced image and the standardized image data are channel-joined. The initial enhanced image and the original normalized image are combined as different channels into a multi-channel input data. The multi-channel data is input into the generator of the conditional generative adversarial network in the second stage. The generator in the second stage refines the image enhancement effect and generates a more refined enhanced image. In this stage, the generator is able to adjust the local contrast, details and color balance of the image so that the enhanced image is closer to the lighting conditions in the natural scene. Global contrast adaptive histogram equalization is performed on the generated refined enhanced image. Histogram equalization is a technique for enhancing image contrast. It adjusts the brightness distribution of pixels in the image to make the bright and dark parts of the image more balanced. In global contrast adaptive histogram equalization, the contrast is dynamically adjusted according to the local features of different areas of the image, so as to avoid the loss of details caused by over-enhancement and generate an image with optimized contrast. In order to evaluate the enhancement effect, the structural similarity index (SSIM) and peak signal-to-noise ratio (PSNR) of the image are calculated. SSIM measures the structural similarity between two images, and its value range is [−1, 1]. The closer it is to 1, the higher the structural similarity of the images. PSNR is used to measure the difference between the enhanced image and the original image. The higher the PSNR value, the more details in the original image are retained in the enhanced image. After completing the image quality assessment, the finely enhanced image is input into the discriminator of the second-stage conditional generative adversarial network. The task of the discriminator is to determine whether the input image is a real image. Through the trained model, the discriminator can give a realism score for the finely enhanced image, reflecting the degree of similarity between the enhanced image and the natural scene image. The adversarial loss is calculated based on the realism score, and the adversarial loss value measures the performance of the generator in generating high-quality images. The adversarial loss formula can be expressed as:

[0062] ;

[0063] in, Represents the discriminator for the real image Ratings, An image representing the output of the generator , is random noise. The loss function optimizes the performance of the generator by maximizing the score of the discriminator for the real image and minimizing the score of the discriminator for the generated image. Calculate the pixel loss value for the fine enhanced image and the standardized image data. The pixel loss value measures the difference between the two images at the pixel level. For example, the mean square error (MSE) can be used to measure the error between the two images at each pixel position. Extract the perceptual loss value. The perceptual loss focuses more on the difference in high-level semantic features of the image and can capture the subtle differences in style and texture of the image. The adversarial loss value, pixel loss value and perceptual loss value are weighted and summed based on the preset weights to obtain the joint loss function value. The joint loss value can comprehensively reflect the difference between the image at different levels and provide guidance for optimizing the generator and discriminator. Finally, the parameters of the first and second stage conditional generative adversarial networks are updated by backpropagating the joint loss value. In this way, the generator can gradually improve its ability to generate high-quality enhanced images, while the discriminator is also continuously improving its ability to distinguish between real images and generated images. After updating the network parameters, based on the optimized network parameters, the standardized image data is processed in two stages to finally generate the target enhanced image.

[0064] In one example, target detection is performed on the target enhanced image to obtain the target bounding box and category information, and based on the target bounding box, a single convolutional attention module is used to extract deep feature vectors and visual appearance features to obtain a multimodal feature representation, including:

[0065] Perform multi-scale pyramid transformation on the target enhanced image to obtain feature maps of multiple scales, and perform target detection on the feature maps of multiple scales to obtain the initial bounding box and category probability distribution;

[0066] Based on the initial bounding box and category probability distribution, a non-maximum suppression operation is performed to obtain the target bounding box and category information, and the coordinates of the target bounding box are normalized to obtain a standardized target bounding box;

[0067] Map the standardized target bounding box to the target enhanced image, extract the image block of the corresponding area, obtain the target image block, and input the target image block into the single convolution attention module for processing to obtain the attention weight map;

[0068] The target image block is weighted based on the attention weight map to obtain the attention-enhanced target image block, and multi-layer convolution feature extraction is performed on the attention-enhanced target image block to obtain a deep feature vector;

[0069] The color histogram, HOG feature and LBP texture feature are calculated for the attention-enhanced target image block to obtain the visual appearance feature, and the deep feature vector is concatenated with the visual appearance feature to obtain a multimodal feature representation.

[0070] In this example, a multi-scale pyramid transform is performed on the target enhanced image to extract features of different scales in the image. Image pyramid is an image processing method that constructs image representations at different levels by gradually reducing the resolution of the image, which helps capture targets of different sizes. In the multi-scale pyramid transform, by downsampling the image step by step, multiple image feature maps of different scales are obtained to reflect the details and overall structure in the image. Target detection is performed on feature maps of multiple scales. The feature maps are analyzed through models such as convolutional neural networks, and the initial bounding box and category probability distribution of the target are generated. The category probability distribution indicates the possibility that each target belongs to different categories, and the bounding box defines the specific location of the target in the image. For example, if the detected object is a person, the category probability distribution may include the possibility of multiple categories such as "person", "animal", "object", etc., and the bounding box identifies the location of the person. After obtaining the initial bounding box, in order to remove redundant and overlapping bounding boxes, a non-maximum suppression (NMS) operation is performed. NMS is a classic post-processing algorithm used to eliminate redundant bounding boxes in target detection. Assume that there are two bounding boxes and If the overlap between them is too high (for example, exceeding a certain threshold), only the bounding box with the higher score is retained, and the one with the lower score is removed. This overlap is usually measured by the intersection-over-union ratio, as follows:

[0071] ;

[0072] in, represents the overlapping area of ​​two bounding boxes, Represents the union area of ​​two bounding boxes. If IoU is greater than a certain threshold (e.g., 0.5), the bounding boxes with larger overlap are deleted and the boxes with higher scores are retained. In this way, NMS ensures that each target has only one optimal bounding box description, eliminating duplicate detections. After performing non-maximum suppression, the coordinates of the remaining target bounding boxes are normalized, and the coordinates of the bounding boxes are scaled to a uniform scale, such as normalizing the coordinates of all bounding boxes to the range of [0, 1], thereby eliminating the impact of image size changes on target detection accuracy and making subsequent processing more stable. The normalized target bounding box is mapped back to the target enhanced image, and the corresponding image area, i.e., the target image patch, is extracted from it. These image patches contain the local area of ​​the detected target. The image patch is input into a single convolutional attention module. The attention mechanism is a mechanism that imitates the human visual system and can automatically detect important areas in an image. Through the single convolutional attention module, an attention weight map is generated, which assigns a weight value to each pixel to reflect the importance of the pixel. For more critical areas, the weight is higher, while for unimportant areas, the weight is lower. Based on the generated attention weight map, the target image block is weighted to obtain the target image block with enhanced attention. Multi-layer convolutional feature extraction is performed on the image block with enhanced attention. Multi-layer convolutional neural network can extract deep features of the image layer by layer. These convolutional features constitute deep feature vectors, which represent the global and local information of the target. Some classic visual appearance features are extracted from the target image block. The color histogram is calculated. The color histogram reflects the distribution of different colors in the image and can help the system identify the color characteristics of the target. The directional gradient histogram (HOG) feature is calculated. The HOG feature can capture the edge and contour information of the target by analyzing the gradient direction of the image, which is suitable for object detection and shape recognition. The local binary pattern (LBP) feature is calculated. The LBP feature can reflect the texture information of the image, especially for facial recognition and texture classification tasks. The deep feature vector is spliced ​​with the color histogram, HOG feature and LBP feature to obtain the final multimodal feature representation. The multimodal feature combines the semantic information, color information, shape information and texture information of the target.

[0073] In one example, based on the multimodal feature representation, template matching is performed on the detected human target to obtain an initial recognition result, including:

[0074] Normalize the multimodal feature representation to obtain a standardized feature vector, and build a local sensitive hash index based on the standardized feature vector to obtain a fast retrieval structure;

[0075] Extracting multiple template features from a pre-built family member feature library, normalizing the multiple template features to obtain a standardized template feature set, and performing a nearest neighbor search in a fast retrieval structure based on the standardized template feature set to obtain a candidate matching pair;

[0076] Calculate the cosine similarity of the candidate matching pairs to obtain a similarity score, and set an adaptive threshold based on the similarity score to filter the matching pairs below the threshold to obtain the initial matching result;

[0077] Verify the geometric consistency of the initial matching results to obtain the target matching results, and calculate the weighted voting scores based on the target matching results to obtain the identity candidate list;

[0078] For each candidate identity in the identity candidate list, extract the corresponding historical recognition record, construct the temporal consistency constraint, obtain the temporal weighted score, and merge the temporal weighted score with the matching score of the current frame to obtain a comprehensive score;

[0079] The candidate identities are ranked based on the comprehensive scores, and the identity with the highest score is selected as the initial recognition result.

[0080] In this example, the multimodal feature representation is normalized. The multimodal feature representation is usually composed of multiple features, including deep convolutional features, color histograms, HOG features, and LBP features. These features usually have different scales and ranges. In order to ensure that they can equally affect the similarity calculation in subsequent calculations, these features are normalized to the same range, usually between [0, 1] or [−1, 1]. The normalized feature vector is represented as a standardized feature vector. A locality-sensitive hashing (LSH) index is constructed based on the standardized feature vector. LSH is a fast retrieval technology commonly used for high-dimensional data. By mapping similar data points to the same hash bucket, LSH can greatly improve the retrieval efficiency. LSH constructs multiple hash functions by performing multiple random projections on the standardized feature vector. Each hash function can map similar feature vectors to the same hash bucket to form a fast retrieval structure. Multiple template features are extracted from the pre-built family member feature library. These template features represent the feature information of family members known to the system. Normalize the template features to ensure that the template features and input features have the same scale and obtain a standardized template feature set. Based on the standardized template feature set, perform nearest neighbor search in the fast retrieval structure constructed by LSH, find the template features that are most similar to the input features from the feature library, and form candidate matching pairs. Since LSH maps similar feature vectors to the same hash bucket, the nearest neighbor search only needs to be performed in these hash buckets, reducing the search space and improving the retrieval speed. Calculate the cosine similarity for the candidate matching pairs. Cosine similarity is used to measure the angle between two vectors. The value range is between [-1, 1]. The closer the value is to 1, the more similar the two vectors are. The formula for cosine similarity is:

[0081] ;

[0082] in, and Represent two feature vectors to be compared, represents the dot product of two vectors, and are the moduli of two vectors respectively. The calculated similarity score is used to measure the matching degree between the input feature and the template feature. Based on the similarity score, an adaptive threshold is set to filter out the matching pairs below the threshold to obtain the initial matching result. The initial matching result is verified for geometric consistency to ensure that the matching pairs are also reasonable in spatial position. For example, in facial recognition, if the facial key points of a matching pair are inconsistent with the known template features in space, the matching pair may be a mismatch. Through geometric consistency verification, unreasonable matching pairs are excluded to obtain more reliable target matching results. The weighted voting score is calculated based on the target matching result. The weighted voting mechanism is used to synthesize the results of multiple matching pairs to generate an identity candidate list, where the score of each candidate identity is the weighted sum of the scores of multiple matching pairs. For each candidate identity in the identity candidate list, its corresponding historical recognition record is extracted to construct a temporal consistency constraint. The temporal consistency constraint is based on the historical recognition trajectory of the target to ensure that the recognition result of the current frame is consistent with the behavior trajectory of the target. For example, if an identity has been identified as a family member in multiple previous frames, the system will give priority to the identity still being the family member in the current frame. By analyzing the historical records, the temporal weighted score of each candidate identity is calculated to reflect its stability in the historical records. The temporal weighted score is combined with the matching score of the current frame to generate a comprehensive score. All candidate identities are sorted based on the comprehensive score, and the identity with the highest score is selected as the initial recognition result.

[0083] In one example, multiple targets are tracked based on the initial recognition results and the target bounding box to obtain the target trajectory and predicted position, including:

[0084] Data association is performed on the initial recognition result and the target bounding box, and the Hungarian algorithm is used to match the current frame target with the historical trajectory to obtain a matching pair list. Multiple volumetric Kalman filters are initialized based on the matching pair list, and each filter corresponds to a tracking target to obtain an initial filter set.

[0085] Perform strong tracking of multiple fading factors on each filter in the initial filter set, introduce fading factors in the time update and measurement update stages, obtain the corrected state estimate and error covariance, and calculate the new information sequence based on the corrected state estimate and error covariance to obtain the model matching index;

[0086] The Markov transfer matrix in the interactive multi-model algorithm is dynamically adjusted according to the model matching index to obtain the adaptive transfer probability, and multiple motion models are probability-weighted fused based on the adaptive transfer probability to obtain the mixed state estimation and mixed error covariance;

[0087] Perform singular value decomposition on the mixed state estimation to obtain a stable state estimation result, and predict the position and speed of the target in the next frame based on the stable state estimation result to obtain the target prediction state;

[0088] Compare the target prediction state with the actual observation value, calculate the prediction error, obtain the anomaly detection index, and judge whether the target is occluded or moving quickly based on the anomaly detection index. If so, trigger the target re-identification module to obtain the updated target identity information;

[0089] The steady-state estimation results of multiple consecutive frames are smoothed to obtain the target trajectory, and the motion features are extracted based on the target trajectory, including speed, acceleration and motion direction, to obtain the target motion mode, and the target motion mode is associated with the updated target identity information to obtain the target trajectory and predicted position.

[0090] In this example, the Hungarian algorithm is used to match the target in the current frame with the historical trajectory. The Hungarian algorithm finds the best matching pair by minimizing the sum of the cost matrices. Assume that the cost matrix Elements in Indicates the target and historical trajectory The distance or similarity between them, then the Hungarian algorithm solves the following minimization problem:

[0091] ;

[0092] in, Indicates the target Whether with trajectory If , then the target and trajectory After obtaining the matching pair list, multiple volumetric Kalman filters are initialized, each of which is responsible for tracking the state of a target. The Kalman filter mainly recursively estimates the state of the target through the state transfer equation and the observation equation. The time update equation of the Kalman filter (prediction stage) can be expressed as:

[0093] ;

[0094] in, is the predicted state, is the state estimate at the previous moment, is the state transition matrix, is the control input matrix, is the control input. At the completion time, the predicted state is corrected by observing the update equation (correction phase), which is:

[0095] ;

[0096] in, is the Kalman gain matrix, is the current observation value, is the observation matrix, is the predicted state after time update. Kalman gain Determines the weight of the current observation in the updated estimated state, the formula is:

[0097] ;

[0098] in, is the forecast error covariance matrix, is the observation noise covariance matrix. Through the gain matrix, the previous predicted state is corrected according to the observed value to obtain a more accurate state estimate. In order to improve the filter's adaptability to complex target motion, multiple fading factors are introduced for strong tracking. The role of the fading factor is to reduce the impact of historical data on the current estimate, so that the filter can adapt to changes in the target motion state more quickly. As the tracking progresses, multiple different motion models, such as uniform motion and accelerated motion models, are processed through the interactive multi-model algorithm. The interactive multi-model algorithm dynamically adjusts the model weights by weighted fusion of the state estimates of these models, specifically:

[0099] ;

[0100] in, It is The state estimation of the model, It is The weight of the model, is the total number of models. Weight It is updated based on the match between the current motion state of the target and the model. The predicted target state is compared with the actual observation value, the prediction error is calculated, and the anomaly detection module is used to determine whether the target is occluded or moving rapidly. If the system detects an anomaly (such as a large prediction error), the target re-identification module is triggered to update the target identity information. The re-identification module re-identifies the target based on appearance features or other contextual information to ensure that the system can continue to track the target even if it temporarily leaves the camera field of view. The state estimation of multiple consecutive frames is smoothed to obtain a smooth target trajectory. Smoothing reduces trajectory jitter caused by noise, ensuring a more stable and coherent trajectory. The target's motion pattern is constructed by extracting motion features such as speed, acceleration, and direction of motion from the smoothed trajectory. The target's motion pattern provides in-depth information about the target's behavior, allowing the system to determine the target's activity state and intention. The target motion pattern is associated with the updated target identity information to generate a complete target trajectory and predicted position.

[0101] In one example, a spatiotemporal behavior model is constructed based on the target trajectory and predicted position, and a long short-term memory network is used for sequence processing to obtain a behavior feature sequence. The behavior feature sequence is then subjected to anomaly detection and rule matching to obtain event detection results, including:

[0102] The target trajectory is sampled in time and space to obtain a discretized trajectory sequence, and the motion features between adjacent time and space points are calculated based on the discretized trajectory sequence to obtain a motion feature sequence, and the motion feature sequence is combined with the predicted position to construct the initial time and space behavior feature;

[0103] Perform sliding window processing on the initial spatiotemporal behavior features to obtain multiple time segments, perform statistical analysis on the features in each time segment to obtain a statistical feature vector, and concatenate the statistical feature vector with the initial spatiotemporal behavior features to obtain enhanced spatiotemporal behavior features;

[0104] The enhanced spatiotemporal behavior features are input into the pre-trained long short-term memory network, the temporal dependency is modeled, and a hidden state sequence is obtained. The attention mechanism is calculated on the hidden state sequence to obtain an attention-weighted hidden state sequence. The attention-weighted hidden state sequence is mapped to the feature space through a fully connected layer to obtain a behavior feature sequence.

[0105] The behavior feature sequence is input into the support vector machine model, and the distance from the sample to the decision boundary of the input behavior feature sequence is calculated to obtain an abnormal score, and the abnormal score is compared with the preset threshold. If it exceeds the threshold, it is marked as abnormal behavior to obtain the initial anomaly detection result;

[0106] Obtain a rule base, which contains multiple predefined event patterns and corresponding trigger conditions, and perform rule matching on the behavior feature sequence to check whether the trigger conditions in the rule base are met, obtain the rule matching result, and fuse the rule matching result with the initial anomaly detection result to obtain the event type determination result;

[0107] Based on the event type determination result, the preset event response strategy library is queried, the corresponding response action is selected, the event response instruction is obtained, and the event response instruction is combined with the event type determination result to generate the event detection result.

[0108] In this example, the continuous self-labeled trajectory is sampled in time and space and discretized into a trajectory sequence to reduce the computational complexity and extract key information points. The spatiotemporal sampling constructs a discrete trajectory by selecting representative points in time and space. Each discrete point contains the position and velocity information of the target at a specific time. Assume that the target trajectory is in time Upsampled, discretized trajectory sequence Represents the target at each time point Location Through this discretization process, the complexity of the continuous trajectory is simplified. Based on the discretized trajectory sequence, the motion features between adjacent time and space points are calculated. The motion features include the speed, acceleration, and angular velocity of the target, which can reflect the motion changes of the target in each time period. For example, the adjacent time and space points and The speed between can be calculated by the following formula:

[0109] ;

[0110] in, Indicates time speed, is the time interval, and are the coordinates of two adjacent sampling points respectively. Through calculation, a motion feature sequence is generated, which describes the motion state of the self-marker in different time periods. The motion feature sequence is combined with the predicted position of the target to construct the initial spatiotemporal behavior feature, which integrates the current motion state and possible future position of the target, so as to more comprehensively describe the behavior of the target. The initial spatiotemporal behavior feature is processed by sliding window. The sliding window technology divides the trajectory sequence into multiple time segments according to a fixed-size window, and each window contains the behavior features in a continuous time. In each time segment, statistical analysis is performed to extract the statistical feature vector. For example, the average speed, maximum acceleration and other statistical values ​​in the time segment are calculated. The statistical feature vector is spliced ​​with the initial spatiotemporal behavior feature to obtain the enhanced spatiotemporal behavior feature. The enhanced spatiotemporal behavior feature is input into the pre-trained long short-term memory network (LSTM). LSTM can model temporal dependencies and capture the dynamic changes of the target behavior in time. The LSTM network memorizes or forgets the behavior features of the historical time period through its gating mechanism (such as forget gate, input gate and output gate) to generate the hidden state sequence of the target. The hidden state sequence represents the internal state of the target at each moment and is a comprehensive representation of the target behavior in the time dimension. The attention mechanism is applied to the hidden state sequence. The attention mechanism calculates the importance weight of each time point and assigns higher weights to those time points that have a greater impact on the behavior analysis, thereby generating an attention-weighted hidden state sequence. The weighted hidden state sequence is mapped to the feature space through a fully connected layer to obtain the final behavior feature sequence. The behavior feature sequence is input into the support vector machine (SVM) model for anomaly detection. The SVM model can calculate the distance from the input behavior feature sequence to the decision boundary. The larger the distance, the more likely the behavior is to be abnormal. The decision function of SVM can be expressed as:

[0111] ;

[0112] in, is the weight vector, is the input behavioral feature sequence, is a bias term. By calculating the distance from the sample to the decision boundary, the anomaly score is obtained, and the anomaly score is compared with the preset threshold. If the anomaly score exceeds the threshold, the behavior is marked as abnormal and the initial anomaly detection result is generated. Predefined event patterns and corresponding trigger conditions are extracted from the rule base. These rules are usually defined according to specific behavior patterns. Rule matching is performed on the behavior feature sequence to check whether the target behavior meets the trigger conditions in the rule base. If so, a rule matching result is generated. The rule matching result is merged with the initial anomaly detection result to obtain a more accurate event type determination result. After the event type determination result is generated, the preset event response strategy library is queried to select the response action corresponding to the current event type. For example, if the system determines that someone has entered a sensitive area of ​​the home, the response action may be to issue an alarm or send a notification to the user. The system generates an event response instruction and combines it with the event type determination result to generate the final event detection result.

[0113] Reference Figure 2 , this embodiment provides a smart home detection and management device, including:

[0114] The acquisition module 1 is used to acquire initial image data through a smart home device with a camera module and perform preprocessing to obtain standardized image data and target metadata;

[0115] Processing module 2, used for performing image enhancement processing and joint loss optimization using a dual conditional generative adversarial network based on the standardized image data and target metadata to obtain a target enhanced image;

[0116] Extraction module 3, used to perform target detection on the target enhanced image, obtain the target bounding box and category information, and extract deep feature vectors and visual appearance features based on the target bounding box using a single convolutional attention module to obtain a multimodal feature representation;

[0117] A matching module 4 is used to perform template matching on the detected human target based on the multimodal feature representation to obtain an initial recognition result;

[0118] Tracking module 5, used to perform multi-target tracking on the initial recognition results and target bounding boxes to obtain target trajectories and predicted positions;

[0119] Detection module 6 is used to build a spatiotemporal behavior model based on the target trajectory and predicted position, use a long short-term memory network to perform sequence processing to obtain a behavior feature sequence, and perform anomaly detection and rule matching on the behavior feature sequence to obtain an event detection result.

[0120] In this embodiment, for the specific implementation of each unit in the above device embodiment, please refer to the above method embodiment, which will not be repeated here.

[0121] Reference Figure 3 In an embodiment of the present invention, a computer device is also provided. The computer device may be a server, and its internal structure may be as follows: Figure 3 As shown. The computer device includes a processor, a memory, a display screen, an input device, a network interface and a database connected through a system bus. Among them, the processor designed by the computer is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store the corresponding data in this embodiment. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the above method is implemented.

[0122] Those skilled in the art will understand that Figure 3 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present invention, and does not constitute a limitation on the computer device to which the solution of the present invention is applied.

[0123] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the above method is implemented. It can be understood that the computer-readable storage medium in this embodiment can be a volatile readable storage medium or a non-volatile readable storage medium.

[0124] In summary, a smart home detection and management method is proposed in an embodiment of the present invention.

[0125] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media provided by the present invention and used in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double-speed data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM.

[0126] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the presence of other identical elements in the process, device, article or method including the element.

[0127] The above description is only a preferred embodiment of the present invention, and does not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation based on the contents of the present invention description and drawings, or direct or indirect application in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A smart home detection and management method, characterized in that: The following steps are involved: Initial image data is collected and preprocessed by a smart home device with a camera module to obtain standardized image data and target metadata; Based on the standardized image data and the target metadata, a dual conditional generative adversarial network is used to perform image enhancement processing and joint loss optimization to obtain a target enhanced image; Specifically, the method includes: performing brightness analysis on the standardized image data to obtain a brightness distribution map, and determining an image enhancement strategy based on the brightness distribution map and the light intensity information in the target metadata to obtain an enhancement parameter set; inputting the standardized image data into a generator of a first-stage conditional generative adversarial network, and adjusting the convolutional layer weights of the generator based on the enhancement parameter set to obtain an initial enhanced image; performing channel splicing on the initial enhanced image and the standardized image data to obtain multi-channel input data, and inputting the multi-channel input data into a generator of a second-stage conditional generative adversarial network to obtain a refined enhanced image; performing global contrast adaptive histogram equalization on the refined enhanced image to obtain a contrast optimized image, and calculating a structural similarity index and a peak value based on the contrast optimized image. signal-to-noise ratio, and obtain an image quality assessment index; input the fine enhanced image into the discriminator of the second-stage conditional generative adversarial network to obtain a authenticity score, and calculate the adversarial loss based on the authenticity score to obtain an adversarial loss value; calculate the pixel loss value for the fine enhanced image and the standardized image data, and extract the perceptual loss value for the fine enhanced image and the standardized image data; perform weighted summation of the adversarial loss value, the pixel loss value, and the perceptual loss value based on preset weights to obtain a joint loss function value, and back-propagate the joint loss function value to update the parameters of the first-stage and second-stage conditional generative adversarial networks to obtain optimized network parameters; based on the optimized network parameters, re-perform two-stage processing on the standardized image data to obtain a target enhanced image; Performing target detection on the target enhanced image to obtain a target bounding box and category information, and based on the target bounding box, using a single convolutional attention module to extract deep feature vectors and visual appearance features to obtain a multimodal feature representation; Based on the multimodal feature representation, template matching is performed on the detected human target to obtain an initial recognition result; specifically, the method includes: normalizing the multimodal feature representation to obtain a standardized feature vector, and constructing a local sensitive hash index based on the standardized feature vector to obtain a fast retrieval structure; extracting multiple template features from a pre-constructed family member feature library, normalizing the multiple template features to obtain a standardized template feature set, and performing a nearest neighbor search in the fast retrieval structure based on the standardized template feature set to obtain a candidate matching pair; calculating the cosine similarity of the candidate matching pair to obtain a similarity score , and set an adaptive threshold based on the similarity score, filter the matching pairs below the threshold, and obtain an initial matching result; perform geometric consistency verification on the initial matching result to obtain a target matching result, and calculate a weighted voting score based on the target matching result to obtain an identity candidate list; for each candidate identity in the identity candidate list, extract the corresponding historical recognition record, construct a temporal consistency constraint, obtain a temporal weighted score, and fuse the temporal weighted score with the matching score of the current frame to obtain a comprehensive score; sort the candidate identities based on the comprehensive score, and select the identity with the highest score as the initial recognition result; Perform multi-target tracking on the initial recognition result and the target bounding box to obtain the target trajectory and predicted position; specifically including: performing data association on the initial recognition result and the target bounding box, using the Hungarian algorithm to match the current frame target with the historical trajectory to obtain a matching pair list, and initializing multiple volumetric Kalman filters based on the matching pair list, each filter corresponds to a tracking target, and obtains an initial filter set; performing multiple fading factor strong tracking on each filter in the initial filter set, introducing fading factors in the time update and measurement update stages to obtain a corrected state estimate and error covariance, and calculating a new information sequence based on the corrected state estimate and error covariance to obtain a model matching index; dynamically adjusting the Markov transfer matrix in the interactive multi-model algorithm according to the model matching index to obtain an adaptive transition probability, and based on the adaptive The transfer probability performs probability-weighted fusion on multiple motion models to obtain a mixed state estimate and a mixed error covariance; the mixed state estimate is subjected to singular value decomposition to obtain a stable state estimate result, and the position and speed of the target in the next frame are predicted based on the stable state estimate result to obtain a target predicted state; the target predicted state is compared with the actual observation value, the prediction error is calculated, an anomaly detection index is obtained, and based on the anomaly detection index, it is determined whether the target is occluded or moves rapidly, and if so, the target re-identification module is triggered to obtain updated target identity information; the stable state estimate results of multiple consecutive frames are smoothed to obtain a target trajectory, and motion features, including speed, acceleration and motion direction, are extracted based on the target trajectory to obtain a target motion pattern, and the target motion pattern is associated with the updated target identity information to obtain a target trajectory and a predicted position; A spatiotemporal behavior model is constructed based on the target trajectory and the predicted position, and a long short-term memory network is used for sequence processing to obtain a behavior feature sequence, and the behavior feature sequence is subjected to anomaly detection and rule matching to obtain an event detection result; specifically, the following steps are performed: spatiotemporally sampling the target trajectory to obtain a discretized trajectory sequence, and motion features between adjacent spatiotemporal points are calculated based on the discretized trajectory sequence to obtain a motion feature sequence, and the motion feature sequence is combined with the predicted position to construct an initial spatiotemporal behavior feature; sliding window processing is performed on the initial spatiotemporal behavior feature to obtain multiple time segments, and statistical analysis is performed on the features within each time segment to obtain a statistical feature vector, and the statistical feature vector is concatenated with the initial spatiotemporal behavior feature to obtain an enhanced spatiotemporal behavior feature; the enhanced spatiotemporal behavior feature is input into a pre-trained long short-term memory network to model the temporal dependency relationship to obtain a hidden state sequence, and an attention machine is used to perform the attention machine matching on the hidden state sequence. A method for detecting an abnormality in a scene is disclosed in claim 14. The method comprises the following steps: calculating a weighted hidden state sequence of attention, mapping the weighted hidden state sequence of attention to the feature space through a fully connected layer, and obtaining a behavior feature sequence; inputting the behavior feature sequence into a support vector machine model, calculating the distance from the sample to the decision boundary of the input behavior feature sequence, obtaining an abnormal score, and comparing the abnormal score with a preset threshold value. If the threshold value is exceeded, the abnormal behavior is marked as abnormal behavior, and an initial abnormality detection result is obtained; obtaining a rule library, which includes multiple predefined event patterns and corresponding trigger conditions, and performing rule matching on the behavior feature sequence to check whether the trigger conditions in the rule library are met to obtain a rule matching result, and merging the rule matching result with the initial abnormality detection result to obtain an event type determination result; querying a preset event response strategy library based on the event type determination result, selecting a corresponding response action, obtaining an event response instruction, and combining the event response instruction with the event type determination result to generate an event detection result.

2. The smart home detection and management method according to claim 1, characterized in that: The method of collecting initial image data and preprocessing the data by using a smart home device with a camera module to obtain standardized image data and target metadata includes: Collecting initial image data through a smart home device with a camera module, and performing bidirectional filtering on the initial image data to obtain denoised image data; Performing perspective transformation matrix analysis on the denoised image data to obtain geometrically corrected image data; Performing edge detection on the geometrically corrected image data to obtain an edge map, and performing Hough transform based on the edge map to obtain target straight line and curve features; Performing image segmentation based on the target straight line and curve features to obtain multiple candidate regions of interest, and performing region growing analysis on the candidate regions of interest to obtain finely segmented regions of interest; Obtaining the current timestamp and camera parameter information, and associating them with the finely segmented region of interest to obtain initial metadata, and measuring the ambient light intensity based on a built-in light sensor, adding the measurement result to the initial metadata to obtain target metadata; The finely segmented region of interest is scale normalized and pixel value normalized to obtain standardized image data.

3. The smart home detection and management method according to claim 1, characterized in that: The target enhanced image is subjected to target detection to obtain a target bounding box and category information, and based on the target bounding box, a single convolutional attention module is used to extract deep feature vectors and visual appearance features to obtain a multimodal feature representation, including: Performing a multi-scale pyramid transformation on the target enhanced image to obtain feature maps of multiple scales, and performing target detection on the feature maps of multiple scales to obtain an initial bounding box and category probability distribution; Performing a non-maximum suppression operation based on the initial bounding box and the category probability distribution to obtain a target bounding box and category information, and performing coordinate normalization processing on the target bounding box to obtain a standardized target bounding box; Mapping the standardized target bounding box onto the target enhanced image, extracting image blocks of corresponding areas to obtain target image blocks, and inputting the target image blocks into a single convolution attention module for processing to obtain an attention weight map; Weighting the target image block based on the attention weight map to obtain an attention-enhanced target image block, and performing multi-layer convolution feature extraction on the attention-enhanced target image block to obtain a deep feature vector; A color histogram, HOG features and LBP texture features are calculated for the attention-enhanced target image block to obtain a visual appearance feature, and the deep feature vector is concatenated with the visual appearance feature to obtain a multimodal feature representation.

4. A smart home detection and management device, characterized in that: For executing the steps of the method according to any one of claims 1 to 3, the device comprises: An acquisition module is used to acquire initial image data through a smart home device with a camera module and perform preprocessing to obtain standardized image data and target metadata; A processing module, configured to perform image enhancement processing and joint loss optimization using a dual conditional generative adversarial network based on the standardized image data and the target metadata to obtain a target enhanced image; An extraction module, configured to perform target detection on the target enhanced image to obtain a target bounding box and category information, and based on the target bounding box, use a single convolutional attention module to extract deep feature vectors and visual appearance features to obtain a multimodal feature representation; A matching module, used for performing template matching on the detected human target based on the multimodal feature representation to obtain an initial recognition result; A tracking module, used for performing multi-target tracking on the initial recognition result and the target bounding box to obtain a target trajectory and a predicted position; The detection module is used to build a spatiotemporal behavior model based on the target trajectory and the predicted position, use a long short-term memory network to perform sequence processing to obtain a behavior feature sequence, and perform anomaly detection and rule matching on the behavior feature sequence to obtain an event detection result.

5. A computer device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 3 are implemented.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 3 are implemented.

Images