Visual cat eye intelligent control method and system based on deep learning

Abstract

The invention discloses a visual cat eye intelligent control method and system based on deep learning, and relates to the field of cat eye intelligent control, and the method comprises the following operation steps: S1, multi-modal data fusion collection; s2, behavior pattern recognition; s3, transfer learning and incremental training; and S4, task division and definition: performing deep analysis on the cat eye intelligent control demand tasks, setting a clear input and output interface for each task, and defining association rules among the tasks. According to the visual cat eye intelligent control method and system based on deep learning, when to start the corresponding task can be accurately judged through the clear triggering condition and rule, unnecessary task starting can be effectively avoided, the operation efficiency and accuracy of the cat eye are improved, and the user experience is improved. And a plurality of factors and conditions can be comprehensively considered to trigger corresponding cat eye tasks, so that effective linkage and cooperative work among the tasks are realized, and the processing capability of the cat eye on complex conditions is improved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent control of cat's eyes, and in particular to a method and system for intelligent control of a visualized cat's eye based on deep learning. Background Art

[0002] Smart cat's eye, also known as video doorbell, integrates the functions of cat's eye and doorbell, overcomes the shortcomings and safety hazards of traditional optical cat's eye, and increases the monitoring function. It is currently widely used at the entrances of homes and high-end hotels. It is an upgrade and alternative product to traditional optical cat's eye.

[0003] Traditional smart cat's eyes can only provide simple visual functions, cannot accurately identify targets and behaviors, and will have performance degradation in complex lighting, bad weather and other environments. In complex scenes, such as when multiple people appear at the same time and the background is noisy, the stability and accuracy of the smart cat's eyes cannot be guaranteed. The response accuracy of traditional smart cat's eyes is poor, and they cannot trigger appropriate tasks at the right time. In addition, their triggering conditions are unclear and relatively simple, which makes it easy to cause false triggering and missed triggering, resulting in the control tasks they can solve being relatively simple.

[0004] Therefore, it is necessary to propose a visual cat-eye intelligent control method and system based on deep learning to solve the above problems. Summary of the invention

[0005] The main purpose of the present invention is to provide a visual cat's eye intelligent control method and system based on deep learning, which can effectively solve the problems in the background technology.

[0006] To achieve the above object, the technical solution adopted by the present invention is: The visual cat-eye intelligent control method based on deep learning includes the following steps: S1: Multi-modal data fusion collection. The visual cat's eye is equipped with a visible light camera, a thermal infrared sensor, and an ultrasonic sensor. The data of the three sensors are integrated to obtain more comprehensive and accurate information in front of the door. S2: Behavior pattern recognition, using 3D convolutional neural network to extract features from multimodal data after S1 fusion, to extract spatial features of images and capture changes in data in the time dimension, to integrate and analyze multimodal data collected at different times, to extract the spatiotemporal features of current behavior, and to accurately identify behavior patterns; S3: Transfer learning and incremental training: Use the model pre-trained on large-scale image and video datasets and migrate it to Smart Maoyan. Then, use the data actually collected by Maoyan for incremental training. At this time, the parameters can be continuously updated according to the new data to continuously optimize the recognition ability of the scene in front of the door. S4: Task division and definition: conduct in-depth analysis of the tasks required for Maoyan intelligent control, set clear input and output interfaces for each task, and define the association rules between tasks.

[0007] Preferably, in S1, the visible light camera is used to obtain a high-definition image in front of the door, the thermal infrared sensor is used to sense the heat emitted by the human body, and the ultrasonic sensor is used to detect the distance and moving speed of the object, which specifically includes the following steps: S101: configure the parameters of the sensor, set the shooting resolution, frame rate, and exposure time of the visible light camera; set the temperature measurement range and sampling frequency of the thermal infrared sensor; set the measurement cycle and trigger threshold of the ultrasonic sensor, and synchronize the collection time interval of the sensor data to facilitate the subsequent data fusion processing; S102: Data collection and preprocessing. Each sensor starts to collect data according to the set parameters and time intervals. The visible light camera collects image data in front of the door, the thermal infrared sensor collects temperature field data, and the ultrasonic sensor collects distance and speed data. The collected raw data is preprocessed. S103: Data fusion processing, including feature-level fusion, data-level fusion and decision-level fusion. Feature-level fusion extracts features from the data pre-processed in S102 and then fuses them to extract color, texture and shape features from the image data, temperature distribution features from the temperature field data, and distance change features from the distance and speed data. Data-level fusion uses weighted average to fuse the raw data collected by different sensors and the pre-processed data into a new data set for comprehensive use of the information from each sensor; Decision-level fusion, where each sensor processes and makes decisions independently, and then the decision results are fused; S104: Storing the fused data in the ontology database and the cloud storage platform.

[0008] Preferably, the decision-level fusion specifically includes the following steps: S10301: Independent sensor decision, including visible light camera decision: Identify people, animals, and objects in image data through target detection algorithms, and determine whether they are familiar people, abnormal behaviors, or objects based on their appearance and posture information; Thermal infrared sensor decision: Based on the temperature field data information, analyze whether there is a human heat source, as well as the number, location and movement of the heat source; Ultrasonic sensor decision-making: Based on the distance and speed data measured by ultrasonic waves, it determines whether an object is approaching or moving away from the cat's eye, as well as the object's movement trajectory and speed changes; S10302: Fusion strategy formulation, assigning different weights to the decisions of each sensor based on the performance, reliability and importance factors of each sensor; S10303: Conflict handling and optimization: When the visible light camera identifies a person, but the heat source characteristics detected by the thermal infrared sensor do not match the characteristics of the human body, and the motion trajectory detected by the ultrasonic sensor is different from the normal human motion, a conflict is determined; Analyze the causes of the conflict to determine whether it is caused by sensor failure, environmental interference, or algorithm misjudgment. Find the root cause of the conflict by analyzing sensor data, environmental information, and algorithm processing. Take appropriate optimization and adjustment measures based on the causes of the conflict. S10304: Decision output: make a comprehensive judgment based on the results after processing in S10302 and S10303, and draw a final decision conclusion based on this. Control the cat's eye and its related equipment according to the decision conclusion.

[0009] Preferably, the step S2 specifically includes the following steps: S201: Use a temporal convolutional network to process and fuse the time series features of each sensor modality, learn the relationship between the temporal features of different modalities, and form a comprehensive temporal feature representation; S202: Build a spatiotemporal joint model, use the spatiotemporal convolutional network to take spatial features and temporal features as input, and further mine spatiotemporal joint features; S203: Feature weighting based on attention mechanism: Introduce the attention mechanism to assign different weights to different modal data according to their importance in the spatiotemporal joint features.

[0010] Preferably, the step S3 specifically includes the following steps: S301: Select the original neighborhood and the target neighborhood. The original neighborhood is a field related to Maoyan intelligent control and has a large amount of labeled data. Here, a data set containing person and object recognition is selected. The target field is the Maoyan intelligent control scene. The objects that need to be recognized in the Maoyan intelligent control scene include visitors, couriers, and abnormal behaviors. S302: Select the residual network as the pre-trained model, use the part of the pre-trained model except the last classification layer as the feature extractor, input the data in the cat's eye intelligent control scene into the feature extraction part of the pre-trained model, and extract the corresponding feature representation; S303: retraining and fine-tuning the final classification layer of the pre-trained model on the labeled data of the target domain, and adjusting the parameters of the classification layer according to the specific task of cat-eye intelligent control to adapt it to the classification requirements of the target domain; S304: Evaluate the fine-tuned model using a test dataset of the target domain, and further adjust the model's hyperparameters based on the evaluation results.

[0011] Preferably, the step S4 specifically includes the following steps: S401: The tasks required for Maoyan intelligent control include target detection tasks, behavior analysis tasks, abnormal event detection tasks, biometric recognition tasks, and result fusion and feedback tasks. The target detection task not only requires the recognition of common people, animals, and packages, but also requires the segmentation of people's clothing features and carried items; Behavior analysis tasks: including action sequence analysis and group behavior analysis. Action sequence analysis identifies the action sequence of characters by analyzing continuous video frames, including whether they are ringing the doorbell, knocking on the door, or trying to pick the lock. Group behavior analysis is to analyze the group behavior patterns when multiple people appear in front of the door, including whether there is quarreling or chasing. Abnormal event detection tasks: including sound anomaly detection and light anomaly detection. Sound anomaly detection is used to detect loud noises, violent collisions, glass breaking, and alarms. Light anomaly detection is used to analyze the brightness and contrast of visible light images to detect whether there are abnormal light changes. Biometric recognition tasks: including body temperature detection and facial expression analysis. Body temperature detection is combined with thermal infrared sensor data, using temperature calibration and image registration technology to accurately measure the visitor's body temperature. By establishing a human thermal radiation model to compensate for the influence of ambient temperature, accurate body temperature measurement is achieved to determine whether the visitor has an abnormal physical condition of illness. Facial expression analysis is based on visible light images, using convolutional neural networks and deep learning algorithms to identify the visitor's facial expressions to determine whether they have abnormal expressions of tension, pain, or anger. The result fusion and feedback task is to fuse the multimodal information generated by the above different tasks, build a complete picture of the scene before going out based on the fused information, and make decisions. According to the results of fusion and decision-making, the relevant information is fed back to the user in a network manner; S402: Task association design, clarify the trigger conditions between tasks, and establish information sharing channels between tasks. The target detection task will pass the identified target location and category information to the behavior analysis task, and the behavior analysis task will then feed back the analysis results to the abnormal event detection task.

[0012] Preferably, in S401, the target detection task uses a deep learning-based target detection algorithm to perform target detection, and at the same time, for small targets that may appear in the cat's eye field and partially obscured targets, a feature pyramid network is used to extract and detect data; Action sequence analysis is performed using recurrent neural networks and long short-term memory networks; Sound anomaly detection is based on the audio classification algorithm of Mel-frequency cepstral coefficients and support vector machines. It extracts and classifies the collected sound signals to determine whether there are abnormal sounds. Light anomaly detection determines whether the light is abnormal by setting reasonable light thresholds and change rates combined with image processing algorithms.

[0013] Preferably, in S402, clarifying the triggering conditions between the tasks specifically includes: Target detection-behavior analysis trigger, the triggering conditions are: when the target detection task recognizes a person, and the person's position is within the effective range of cat's eye monitoring, and the target detection finds that the person exists for 5 consecutive frames, then excluding short-term interference factors, the behavior analysis task is triggered. The target detection task passes the person detection information to the behavior analysis task. The behavior analysis task locates the position of the person in the image based on this information and begins to analyze the subsequent actions of the person. The specific formula is: ;in for intersection and comparison; For the detection frame With real frame The area of ​​the intersection of the two boxes, that is, the area of ​​the overlapping part of the two boxes; For the detection frame With real frame The area of ​​the union of the two boxes is the total area of ​​the area covered by the two boxes. The threshold is preset to 0.5. When it is greater than 0.5, the detection is considered accurate and the behavior analysis task is triggered; Behavior analysis - abnormal event detection trigger, the trigger condition is: the behavior analysis task analyzes the character action sequence, if it detects a preset abnormal action, the behavior analysis task will pass the type, duration, and location information of the abnormal action to the abnormal event detection task. The number of occurrences in , the normal action frequency is preset to ,when: , the current action is judged as an abnormal action, and the abnormal event detection task is triggered; Target detection-biometric recognition trigger. The trigger condition is: when the target detection task recognizes a person and the person's position change in three consecutive frames is less than the set threshold, the biometric recognition task is triggered. The target detection task transmits the person's precise position information in the image to the biometric recognition task. Based on this position information, the biometric recognition task accurately locates the person's facial area, conducts facial expression analysis, locates the person's full-body thermal infrared image area, and conducts body temperature detection. Abnormal event detection - multi-task linkage triggering. The triggering conditions are as follows: when the abnormal event detection task detects abnormal sounds or abnormal light changes, it triggers multi-task linkage. On the one hand, it notifies the target detection and behavior analysis tasks to conduct more intensive monitoring and analysis of the current scene; on the other hand, it triggers the result fusion and feedback tasks to send alarm information to the user. The abnormal event detection task passes the type, occurrence time, and duration of the abnormal event to the target detection and behavior analysis tasks for adjusting the analysis strategy. At the same time, it passes the key information of the abnormal event to the result fusion and feedback tasks for generating accurate alarm content. Priority judgment and conflict resolution, including priority setting and conflict resolution. Priority setting: When multiple trigger conditions are met at the same time, set the priority, and set the behavior analysis-abnormal event detection trigger to the highest priority; followed by target detection-biometric recognition trigger; and finally target detection-behavior analysis trigger; Conflict resolution: If a low-priority task has already started executing and a high-priority task is triggered, the low-priority task will be paused and the high-priority task will be processed first. When the high-priority task is completed, decide whether to continue executing the low-priority task based on the recoverability of the task.

[0014] A visual cat-eye intelligent control system based on deep learning includes a data acquisition module, a deep learning model library, a feature extraction module, and a task division and definition module. The data acquisition module is used to collect data information collected by a visible light camera, a thermal infrared sensor, and an ultrasonic sensor, and pre-process the data information; The deep learning model library is used to integrate the data model preprocessed by the data acquisition module, and perform incremental training based on the new data collected by Maoyan in real time; The feature extraction module is used to extract and fuse features of multiple sensor data in spatial and temporal dimensions, to mine potential relationships between data, and to provide feature representation for behavior analysis and abnormal event detection; The task division and definition module is used to divide and define tasks for the needs of cat's eye intelligent control, and is used to divide them into target detection tasks, behavior analysis tasks, abnormal event detection tasks, biometric identification tasks, and result fusion and feedback tasks. A scheduling strategy based on reinforcement learning is adopted to achieve parallel processing and efficient scheduling of target detection, behavior analysis, abnormal event detection, biometric identification, and result fusion and feedback tasks. According to the task trigger mechanism, when the corresponding conditions are met, each task is started and coordinated in a timely manner, and the cat's eye is intelligently controlled based on this.

[0015] Compared with the prior art, the present invention provides a visual cat-eye intelligent control method and system based on deep learning, which has the following beneficial effects: This visual cat-eye intelligent control method and system based on deep learning uses decision-level fusion. In a multi-sensor system, different sensors may produce conflicting decision results due to measurement errors, environmental interference, etc. Decision-level fusion can effectively handle these conflicts through reasonable fusion strategies, integrate the information of each sensor, and make more accurate and reliable decisions.

[0016] The visual cat's eye intelligent control method and system based on deep learning can divide the cat's eye intelligent control task into multiple subtasks such as target detection, behavior analysis, and abnormal event detection through task division and definition. Each subtask has a clear goal and function. This clear division of responsibilities makes the overall design, development, and maintenance easier. Each subtask can be optimized and improved independently without affecting the normal operation of other subtasks. Moreover, each subtask can be executed in parallel on different computing resources, and parallel processing of tasks can be realized, thereby shortening the response time of the system.

[0017] This deep learning-based visual cat's eye intelligent control method and system can accurately determine when to start the corresponding task through clear trigger conditions and rules, avoiding unnecessary task startup, improving the cat's eye's operating efficiency and accuracy, and can comprehensively consider multiple factors and conditions to trigger tasks, achieve effective linkage and collaborative work between tasks, and improve the cat's eye's ability to handle complex situations. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0019] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the present invention is further explained below in conjunction with specific implementation methods.

[0020] Embodiment 1:

[0021] like Figure 1 As shown in the figure, the visual cat-eye intelligent control method based on deep learning includes the following steps: S1: Multi-modal data fusion collection. The visual cat's eye is equipped with a visible light camera, a thermal infrared sensor, and an ultrasonic sensor. The data of the three sensors are integrated to obtain more comprehensive and accurate information in front of the door. The visible light camera is used to obtain a high-definition image in front of the door, the thermal infrared sensor is used to sense the heat emitted by the human body, and the ultrasonic sensor is used to detect the distance and movement speed of the object. The specific steps include: S101: configure the parameters of the sensor, set the shooting resolution, frame rate, and exposure time of the visible light camera; set the temperature measurement range and sampling frequency of the thermal infrared sensor; set the measurement cycle and trigger threshold of the ultrasonic sensor, and synchronize the collection time interval of the sensor data to facilitate the subsequent data fusion processing; S102: Data collection and preprocessing. Each sensor starts to collect data according to the set parameters and time intervals. The visible light camera collects image data in front of the door, the thermal infrared sensor collects temperature field data, and the ultrasonic sensor collects distance and speed data. The collected raw data is preprocessed. The image data collected by the visible light camera is denoised, grayscaled, and normalized; the data collected by the thermal infrared sensor is temperature corrected and bad pixels are repaired; the data collected by the ultrasonic sensor is filtered and outliers are removed; S103: Data fusion processing, including feature-level fusion, data-level fusion and decision-level fusion. Feature-level fusion extracts features from the data pre-processed in S102 and then fuses them to extract color, texture and shape features from the image data, temperature distribution features from the temperature field data, and distance change features from the distance and speed data. Data-level fusion uses weighted average to fuse the raw data collected by different sensors and the pre-processed data into a new data set for comprehensive use of the information from each sensor; Decision-level fusion: Each sensor processes and makes decisions independently, and then fuses these decision results. Specifically, it includes the following steps: S10301: Independent sensor decision, including visible light camera decision: Identify people, animals, and objects in image data through target detection algorithms, and determine whether they are familiar people, abnormal behaviors, or objects based on their appearance and posture information; Thermal infrared sensor decision: Based on the temperature field data information, it analyzes whether there is a human heat source, as well as the number, location and movement of the heat source. At night or in dim light, if a moving human heat source is detected in front of the door through the thermal infrared sensor, it can be judged that someone is approaching the cat's eye. Ultrasonic sensor decision: Based on the distance and speed data measured by ultrasonic waves, it determines whether an object is approaching or moving away from the cat's eye, as well as the object's movement trajectory and speed changes. When it detects that an object is approaching the cat's eye from a distance at a certain speed and the distance is gradually decreasing, it is determined that someone is walking towards the door; S10302: Fusion strategy formulation: Different weights are assigned to the decisions of each sensor based on the performance, reliability and importance of each sensor. During the day when there is sufficient light, the decision weight of the visible light camera is higher; at night when the light is poor, the weight of the thermal infrared sensor is relatively higher. The decision results of each sensor are combined through weighted calculation to obtain an accurate final judgment. S10303: Conflict handling and optimization: When the visible light camera identifies a person, but the heat source characteristics detected by the thermal infrared sensor do not match the characteristics of the human body, and the motion trajectory detected by the ultrasonic sensor is different from the normal human motion, a conflict is determined; Analyze the causes of the conflict to determine whether it is caused by sensor failure, environmental interference, or algorithm misjudgment. Visible light cameras may be interfered by strong light reflections, thermal infrared sensors may be affected by nearby heat sources, and ultrasonic sensors may be interfered by noise. By analyzing sensor data, environmental information, and algorithm processing, find out the root cause of the conflict. Take corresponding optimization and adjustment measures based on the cause of the conflict. If it is a sensor failure, repair or replace the sensor in a timely manner. If it is environmental interference, shielding and filtering can be used to reduce interference. If it is an algorithm misjudgment, the algorithm can be improved and optimized by adding more constraints and feature judgments to improve the accuracy and robustness of the algorithm. Through continuous conflict processing and optimization, the reliability and stability of decision-level fusion can be improved. S10304: Decision output: make a comprehensive judgment based on the results of S10302 and S10303, and draw a final decision conclusion based on it, and control the cat's eye and its related devices according to the decision conclusion; S104: Storing the fused data in the entity database and the cloud storage platform.

[0022] S2: Behavior pattern recognition, using 3D convolutional neural network to extract features from multimodal data after S1 fusion, to extract spatial features of images and capture changes in data in the time dimension, to integrate and analyze multimodal data collected at different times, to extract the spatiotemporal features of current behavior, and to accurately identify behavior patterns, including the following steps: S201: Use a temporal convolutional network to process and fuse the time series features of each sensor modality, learn the relationship between the temporal features of different modalities, and form a comprehensive temporal feature representation; S202: Build a spatiotemporal joint model, use the spatiotemporal convolutional network to take spatial features and temporal features as input, and further mine spatiotemporal joint features; S203: Feature weighting based on attention mechanism. The attention mechanism is introduced to assign different weights to different modal data according to their importance in the spatiotemporal joint features. When judging whether an abnormal event occurs in front of the door, the temperature mutation in the thermal infrared data and the abnormal movement of objects in the visible light image are more important. The attention mechanism can automatically learn these key information and give corresponding weights, thereby highlighting the key features and improving the representation ability of the spatiotemporal joint features for the problem.

[0023] S3: Transfer learning and incremental training: Use the model pre-trained on large-scale image and video datasets and migrate it to Smart Cat Eye. Then, use the data actually collected by Cat Eye for incremental training. At this time, the parameters can be continuously updated according to the new data to continuously optimize the recognition ability of the scene in front of the door. Specifically, the following steps are included: S301: Select the original neighborhood and the target neighborhood. The original neighborhood is a field related to Maoyan intelligent control and has a large amount of labeled data. Here, a data set containing person and object recognition is selected. The target field is the Maoyan intelligent control scene. The objects that need to be recognized in the Maoyan intelligent control scene include visitors, couriers, and abnormal behaviors. S302: Select the residual network as the pre-trained model, use the part of the pre-trained model except the last classification layer as the feature extractor, input the data in the cat's eye intelligent control scene into the feature extraction part of the pre-trained model, and extract the corresponding feature representation; S303: retraining and fine-tuning the final classification layer of the pre-trained model on the labeled data of the target domain, and adjusting the parameters of the classification layer according to the specific task of cat-eye intelligent control to adapt it to the classification requirements of the target domain; S304: Evaluate the fine-tuned model using a test dataset of the target domain, and further adjust the model's hyperparameters based on the evaluation results.

[0024] S4: Task division and definition: Conduct in-depth analysis of the tasks required for Maoyan intelligent control, set clear input and output interfaces for each task, and define the association rules between tasks. The specific steps include: S401: The tasks required for cat's eye intelligent control include target detection tasks, behavior analysis tasks, abnormal event detection tasks, biometric recognition tasks, and result fusion and feedback tasks. The target detection task not only needs to identify common people, animals, and packages, but also needs to segment the clothing features and items carried by people. The target detection task uses a deep learning-based target detection algorithm to detect targets. At the same time, for small targets that may appear in the cat's eye field of view and partially obscured targets, a feature pyramid network is used to extract and detect data; Behavior analysis tasks: including action sequence analysis and group behavior analysis. Action sequence analysis identifies the action sequence of characters by analyzing continuous video frames, including whether they are ringing the doorbell, knocking on the door, or trying to pick the lock. Group behavior analysis is to analyze the group behavior pattern when multiple people appear in front of the door, including whether there is quarreling or chasing. Action sequence analysis uses recurrent neural networks and long short-term memory networks to model the motion trajectory and posture changes of human joints, and learn the patterns and characteristics of different action sequences. Group behavior analysis uses a method based on graph neural networks, taking each person as a node in the graph and the relationship between people as an edge, and understanding group behavior by analyzing the graph structure. Abnormal event detection tasks: including sound anomaly detection and light anomaly detection. Sound anomaly detection is used to detect loud noises, violent collisions, glass breaking, and alarms. Light anomaly detection is used to analyze the brightness and contrast of visible light images to detect whether there are abnormal light changes, including sudden strong light flashes and long periods of abnormally dim light. Sound anomaly detection is based on the audio classification algorithm of Mel frequency cepstral coefficients and support vector machines to extract and classify the collected sound signals to determine whether there are abnormal sounds. Light anomaly detection determines whether the light is abnormal by setting reasonable light thresholds and change rates and combining image processing algorithms. Biometric recognition tasks: including body temperature detection and facial expression analysis. Body temperature detection is combined with thermal infrared sensor data, using temperature calibration and image registration technology to accurately measure the visitor's body temperature. By establishing a human thermal radiation model to compensate for the influence of ambient temperature, accurate body temperature measurement is achieved to determine whether the visitor has an abnormal physical condition of illness. Facial expression analysis is based on visible light images, using convolutional neural networks and deep learning algorithms to identify the visitor's facial expressions to determine whether they have abnormal expressions of tension, pain, or anger. The result fusion and feedback task fuses the multimodal information generated by the above-mentioned different tasks, combines the person identity identified by the target detection task, the behavior state determined by the behavior analysis task, the abnormal sound found by the abnormal event detection task, and the abnormal body temperature detected by the biometric recognition task, and comprehensively analyzes this information. Based on the fused information, a complete portrait of the scene before going out is constructed and a decision is made. This includes judging whether the situation in front of the door is a normal visitor visit, a courier delivery, or an abnormal situation based on the person identity, behavior, and biometric information. This provides a clear decision-making basis for the subsequent feedback link. Based on the results of the fusion and decision-making, the relevant information is fed back to the user via the network. S402: Task association design, clarify the trigger conditions between tasks, and establish information sharing channels between tasks. The target detection task will pass the identified target location and category information to the behavior analysis task, and the behavior analysis task will then feed back the analysis results to the abnormal event detection task. The trigger conditions between tasks are clarified, including: Target detection-behavior analysis trigger, the triggering conditions are: when the target detection task recognizes a person, and the person's position is within the effective range of cat's eye monitoring, and the target detection finds that the person exists for 5 consecutive frames, then excluding short-term interference factors, the behavior analysis task is triggered. The target detection task passes the person detection information to the behavior analysis task. The behavior analysis task locates the position of the person in the image based on this information and begins to analyze the subsequent actions of the person. The specific formula is: ;in for intersection and comparison; For the detection frame With real frame The area of ​​the intersection of the two boxes, that is, the area of ​​the overlapping part of the two boxes; For the detection frame With real box The area of ​​the union of the two boxes is the total area of ​​the area covered by the two boxes. The threshold is preset to 0.5. When it is greater than 0.5, the detection is considered accurate and the behavior analysis task is triggered; Behavior analysis - abnormal event detection trigger, the triggering conditions are: the behavior analysis task analyzes the character action sequence, if it detects a preset abnormal action, such as multiple consecutive attempts to quickly turn the door lock handle, lingering at the door for a long time with large movements, and abnormal interactions in group behavior, then the abnormal event detection task is triggered. The behavior analysis task passes the type, duration, and location area of ​​the abnormal action to the abnormal event detection task. Suppose an action in the behavior analysis is within time The number of occurrences in , the normal action frequency is preset to ,when: , the current action is judged as an abnormal action, and the abnormal event detection task is triggered; Target detection-biometric recognition trigger, the trigger condition is: when the target detection task recognizes a person and the person's position change in 3 consecutive frames is less than the set threshold, the biometric recognition task is triggered to ensure that biometric recognition is performed when the person is relatively stable and improve the accuracy of recognition. The target detection task transmits the precise position information of the person in the image to the biometric recognition task. Based on this position information, the biometric recognition task accurately locates the facial area of ​​the person, conducts facial expression analysis, locates the thermal infrared image area of ​​the person's whole body, and conducts body temperature detection; Abnormal event detection-multi-task linkage trigger, the trigger condition is: when the abnormal event detection task detects abnormal sound and abnormal light changes, it triggers multi-task linkage. On the one hand, it notifies the target detection and behavior analysis tasks to conduct more intensive monitoring and analysis of the current scene; on the other hand, it triggers the result fusion and feedback task to send an alarm message to the user. The abnormal event detection task transmits the type, occurrence time, and duration of the abnormal event to the target detection and behavior analysis tasks for adjusting the analysis strategy, and at the same time transmits the key information of the abnormal event to the result fusion and feedback task for generating accurate alarm content; Priority judgment and conflict resolution, including priority setting and conflict resolution. Priority setting: When multiple trigger conditions are met at the same time, set the priority, and set the behavior analysis-abnormal event detection trigger to the highest priority; followed by target detection-biometric recognition trigger; and finally target detection-behavior analysis trigger; Conflict resolution: If a low-priority task has already started executing and a high-priority task is triggered, the low-priority task will be paused and the high-priority task will be processed first. When the high-priority task is completed, decide whether to continue executing the low-priority task based on the recoverability of the task.

[0025] Embodiment 2:

[0026] The visual cat-eye intelligent control system based on deep learning includes a data acquisition module, a deep learning model library, a feature extraction module, and a task division and definition module. The data acquisition module is used to collect data information collected by visible light cameras, thermal infrared sensors, and ultrasonic sensors, and pre-process them. The deep learning model library is used to integrate the data models preprocessed by the data acquisition module and perform incremental training based on the new data collected by Maoyan in real time; The feature extraction module is used to extract and fuse features of multiple sensor data in spatial and temporal dimensions, to mine potential relationships between data, and to provide feature representation for behavior analysis and abnormal event detection; The task division and definition module is used to divide and define tasks for the needs of cat's eye intelligent control, and divide them into target detection tasks, behavior analysis tasks, abnormal event detection tasks, biometric identification tasks, and result fusion and feedback tasks. It adopts a scheduling strategy based on reinforcement learning to achieve parallel processing and efficient scheduling of target detection, behavior analysis, abnormal event detection, biometric identification, and result fusion and feedback tasks. According to the task trigger mechanism, when the corresponding conditions are met, each task is started and coordinated in time, and the cat's eye is intelligently controlled based on this.

[0027] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A visual cat-eye intelligent control method based on deep learning, characterized by: The steps include: S1: Multi-modal data fusion collection. The visual cat's eye is equipped with a visible light camera, a thermal infrared sensor, and an ultrasonic sensor. The data of the three sensors are integrated to obtain more comprehensive and accurate information in front of the door. S2: Behavior pattern recognition, using 3D convolutional neural network to extract features from multimodal data after S1 fusion, to extract spatial features of images and capture changes in data in the time dimension, to integrate and analyze multimodal data collected at different times, to extract the spatiotemporal features of current behavior, and to accurately identify behavior patterns; S3: Transfer learning and incremental training: Use the model pre-trained on large-scale image and video datasets and migrate it to Smart Maoyan. Then, use the data actually collected by Maoyan for incremental training. At this time, the parameters can be continuously updated according to the new data to continuously optimize the recognition ability of the scene in front of the door. S4: Task division and definition: conduct in-depth analysis of the tasks required for Maoyan intelligent control, set clear input and output interfaces for each task, and define the association rules between tasks.

2. The visual cat's eye intelligent control method based on deep learning according to claim 1 is characterized in that: In S1, the visible light camera is used to obtain a high-definition image in front of the door, the thermal infrared sensor is used to sense the heat emitted by the human body, and the ultrasonic sensor is used to detect the distance and moving speed of the object, which specifically includes the following steps: S101: configure the parameters of the sensor, set the shooting resolution, frame rate, and exposure time of the visible light camera; set the temperature measurement range and sampling frequency of the thermal infrared sensor; set the measurement cycle and trigger threshold of the ultrasonic sensor, and synchronize the collection time interval of the sensor data to facilitate the subsequent data fusion processing; S102: Data collection and preprocessing. Each sensor starts to collect data according to the set parameters and time intervals. The visible light camera collects image data in front of the door, the thermal infrared sensor collects temperature field data, and the ultrasonic sensor collects distance and speed data. The collected raw data is preprocessed. S103: Data fusion processing, including feature-level fusion, data-level fusion and decision-level fusion. Feature-level fusion extracts features from the data pre-processed in S102 and then fuses them to extract color, texture and shape features from the image data, temperature distribution features from the temperature field data, and distance change features from the distance and speed data. Data-level fusion uses weighted average to fuse the raw data collected by different sensors and the pre-processed data into a new data set for comprehensive use of the information from each sensor; Decision-level fusion, where each sensor processes and makes decisions independently, and then the decision results are fused; S104: Storing the fused data in the entity database and the cloud storage platform.

3. The visual cat's eye intelligent control method based on deep learning according to claim 2 is characterized in that: The decision-level fusion specifically includes the following steps: S10301: Independent sensor decision, including visible light camera decision: Identify people, animals, and objects in image data through target detection algorithms, and determine whether they are familiar people, abnormal behaviors, or objects based on their appearance and posture information; Thermal infrared sensor decision: Based on the temperature field data information, analyze whether there is a human heat source, as well as the number, location and movement of the heat source; Ultrasonic sensor decision-making: Based on the distance and speed data measured by ultrasonic waves, it determines whether an object is approaching or moving away from the cat's eye, as well as the object's movement trajectory and speed changes; S10302: Fusion strategy formulation, assigning different weights to the decisions of each sensor based on the performance, reliability and importance factors of each sensor; S10303: Conflict handling and optimization: When the visible light camera identifies a person, but the heat source characteristics detected by the thermal infrared sensor do not match the characteristics of the human body, and the motion trajectory detected by the ultrasonic sensor is different from the normal human motion, a conflict is determined; Analyze the causes of the conflict to determine whether it is caused by sensor failure, environmental interference, or algorithm misjudgment. Find the root cause of the conflict by analyzing sensor data, environmental information, and algorithm processing. Take appropriate optimization and adjustment measures based on the causes of the conflict. S10304: Decision output: make a comprehensive judgment based on the results after processing in S10302 and S10303, and draw a final decision conclusion based on this. Control the cat's eye and its related equipment according to the decision conclusion.

4. The visual cat's eye intelligent control method based on deep learning according to claim 1 is characterized in that: The S2 specifically includes the following steps: S201: Use a temporal convolutional network to process and fuse the time series features of each sensor modality, learn the relationship between the temporal features of different modalities, and form a comprehensive temporal feature representation; S202: Build a spatiotemporal joint model, use the spatiotemporal convolutional network to take spatial features and temporal features as input, and further mine spatiotemporal joint features; S203: Feature weighting based on attention mechanism: Introduce the attention mechanism to assign different weights to different modal data according to their importance in the spatiotemporal joint features.

5. The visual cat's eye intelligent control method based on deep learning according to claim 1 is characterized in that: The S3 specifically includes the following steps: S301: Select the original neighborhood and the target neighborhood. The original neighborhood is a field related to Maoyan intelligent control and has a large amount of labeled data. Here, a data set containing person and object recognition is selected. The target field is the Maoyan intelligent control scene. The objects that need to be recognized in the Maoyan intelligent control scene include visitors, couriers, and abnormal behaviors. S302: Select the residual network as the pre-trained model, use the part of the pre-trained model except the last classification layer as the feature extractor, input the data in the cat's eye intelligent control scene into the feature extraction part of the pre-trained model, and extract the corresponding feature representation; S303: retraining and fine-tuning the final classification layer of the pre-trained model on the labeled data of the target domain, and adjusting the parameters of the classification layer according to the specific task of cat-eye intelligent control to adapt it to the classification requirements of the target domain; S304: Evaluate the fine-tuned model using a test dataset of the target domain, and further adjust the model's hyperparameters based on the evaluation results.

6. The visual cat's eye intelligent control method based on deep learning according to claim 1, characterized in that: The S4 specifically includes the following steps: S401: The tasks required for Maoyan intelligent control include target detection tasks, behavior analysis tasks, abnormal event detection tasks, biometric recognition tasks, and result fusion and feedback tasks. The target detection task not only requires the recognition of common people, animals, and packages, but also requires the segmentation of people's clothing features and carried items; Behavior analysis tasks: including action sequence analysis and group behavior analysis. Action sequence analysis identifies the action sequence of characters by analyzing continuous video frames, including whether they are ringing the doorbell, knocking on the door, or trying to pick the lock. Group behavior analysis is to analyze the group behavior patterns when multiple people appear in front of the door, including whether there is quarreling or chasing. Abnormal event detection tasks: including sound anomaly detection and light anomaly detection. Sound anomaly detection is used to detect loud noises, violent collisions, glass breaking, and alarms. Light anomaly detection is used to analyze the brightness and contrast of visible light images to detect whether there are abnormal light changes. Biometric recognition tasks: including body temperature detection and facial expression analysis. The body temperature detection is combined with thermal infrared sensor data, using temperature calibration and image registration technology to accurately measure the visitor's body temperature. By establishing a human thermal radiation model to compensate for the influence of ambient temperature, accurate body temperature measurement is achieved to determine whether the visitor has an abnormal physical condition of illness. Facial expression analysis is based on visible light images, using convolutional neural networks and deep learning algorithms to identify the visitor's facial expressions to determine whether they have abnormal expressions of tension, pain, or anger. The result fusion and feedback task is to fuse the multimodal information generated by the above different tasks, build a complete picture of the scene before going out based on the fused information, and make decisions. According to the results of fusion and decision-making, the relevant information is fed back to the user in a network manner; S402: Task association design, clarify the trigger conditions between tasks, and establish information sharing channels between tasks. The target detection task will pass the identified target location and category information to the behavior analysis task, and the behavior analysis task will then feed back the analysis results to the abnormal event detection task.

7. The visual cat's eye intelligent control method based on deep learning according to claim 6 is characterized by: In S401, the target detection task uses a deep learning-based target detection algorithm to perform target detection, and at the same time, a feature pyramid network is used to extract and detect data for small targets and partially obscured targets that may appear in the cat's eye field of view; Action sequence analysis is performed using recurrent neural networks and long short-term memory networks; Sound anomaly detection is based on the audio classification algorithm of Mel-frequency cepstral coefficients and support vector machines. It extracts and classifies the collected sound signals to determine whether there are abnormal sounds. Light anomaly detection determines whether the light is abnormal by setting reasonable light thresholds and change rates combined with image processing algorithms.

8. The visual cat's eye intelligent control method based on deep learning according to claim 6 is characterized by: In S402, the triggering conditions between the tasks are clarified to include: Target detection-behavior analysis trigger, the triggering conditions are: when the target detection task recognizes a person, and the person's position is within the effective range of cat's eye monitoring, and the target detection finds that the person exists for 5 consecutive frames, then excluding short-term interference factors, the behavior analysis task is triggered. The target detection task passes the person detection information to the behavior analysis task. The behavior analysis task locates the position of the person in the image based on this information and begins to analyze the subsequent actions of the person. The specific formula is: ;in is the intersection and comparison; For the detection frame With real box The area of ​​the intersection of the two boxes, that is, the area of ​​the overlapping part of the two boxes; For the detection frame With real box The area of ​​the union of the two boxes is the total area of ​​the area covered by the two boxes. The threshold is preset to 0.

5. When it is greater than 0.5, the detection is considered accurate and the behavior analysis task is triggered; Behavior analysis - abnormal event detection trigger, the trigger condition is: the behavior analysis task analyzes the character action sequence, if it detects a preset abnormal action, the behavior analysis task will pass the type, duration, and location information of the abnormal action to the abnormal event detection task. The number of occurrences in , the normal action frequency is preset to ,when: , the current action is judged as an abnormal action, and the abnormal event detection task is triggered; Target detection-biometric recognition trigger. The trigger condition is: when the target detection task recognizes a person and the person's position change in three consecutive frames is less than the set threshold, the biometric recognition task is triggered. The target detection task transmits the person's precise position information in the image to the biometric recognition task. Based on this position information, the biometric recognition task accurately locates the person's facial area, conducts facial expression analysis, locates the person's full-body thermal infrared image area, and conducts body temperature detection. Abnormal event detection - multi-task linkage triggering. The triggering conditions are as follows: when the abnormal event detection task detects abnormal sounds or abnormal light changes, it triggers multi-task linkage. On the one hand, it notifies the target detection and behavior analysis tasks to conduct more intensive monitoring and analysis of the current scene; on the other hand, it triggers the result fusion and feedback tasks to send alarm information to the user. The abnormal event detection task passes the type, occurrence time, and duration of the abnormal event to the target detection and behavior analysis tasks for adjusting the analysis strategy. At the same time, it passes the key information of the abnormal event to the result fusion and feedback tasks for generating accurate alarm content. Priority judgment and conflict resolution, including priority setting and conflict resolution. Priority setting: When multiple trigger conditions are met at the same time, set the priority, and set the behavior analysis-abnormal event detection trigger to the highest priority; followed by target detection-biometric recognition trigger; and finally target detection-behavior analysis trigger; Conflict resolution: If a low-priority task has already started executing and a high-priority task is triggered, the low-priority task will be paused and the high-priority task will be processed first. When the high-priority task is completed, decide whether to continue executing the low-priority task based on the recoverability of the task.

9. A visual cat's eye intelligent control system based on deep learning, using the visual cat's eye intelligent control method based on deep learning as claimed in any one of claims 1 to 8, comprising a data acquisition module, a deep learning model library, a feature extraction module, and a task division and definition module, characterized in that: The data acquisition module is used to collect data information collected by the visible light camera, thermal infrared sensor, and ultrasonic sensor, and pre-process the data information; The deep learning model library is used to integrate the data model preprocessed by the data acquisition module, and perform incremental training based on the new data collected by Maoyan in real time; The feature extraction module is used to extract and fuse features of multiple sensor data in spatial and temporal dimensions, to mine potential relationships between data, and to provide feature representation for behavior analysis and abnormal event detection; The task division and definition module is used to divide and define tasks for the needs of cat's eye intelligent control, and is used to divide them into target detection tasks, behavior analysis tasks, abnormal event detection tasks, biometric identification tasks, and result fusion and feedback tasks. A scheduling strategy based on reinforcement learning is adopted to achieve parallel processing and efficient scheduling of target detection, behavior analysis, abnormal event detection, biometric identification, and result fusion and feedback tasks. According to the task trigger mechanism, when the corresponding conditions are met, each task is started and coordinated in a timely manner, and the cat's eye is intelligently controlled based on this.

Images