Intelligent interactive door control system

Abstract

The invention discloses an intelligent interactive door control system, and relates to the technical field of door control systems. An image acquisition module is used for acquiring image information in front of a door in real time; the space processing module is used for processing and analyzing the collected images and human body movement tracks, automatically identifying the identity of the visitor and judging behaviors; the voice interaction module is used for generating a voice conversation with a visitor; the control module is used for calling the voice interaction module to communicate with the visitor according to the recognition result of the space processing module; according to the invention, active and intelligent noise suppression is realized, and the interaction reliability in a severe acoustic environment is ensured. The core of the system is to break the barriers of vision and hearing, and the system can understand the meaning of speech by performing feature-level fusion on the visual information and the voice acoustic features, so as to make distinct and highly personalized responses. By bypassing strong dependence on a final text recognition result, fusion is creatively carried out on a feature level.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of the door control system, and in particular to an intelligent interactive door control system. BACKGROUND

[0002] With the continuous rise of technology and the wide use of Internet of Things technology, intelligent security has become a key part of modern social life and enterprise management. Traditional security systems are limited by simple physical protection and limited monitoring capabilities, while modern intelligent security systems are moving towards integration, networking, and intelligence, with a greater emphasis on user experience, security effectiveness, and real-time response. As a key facility for building entrances, the improvement of the intelligence and interactive experience of doors is increasingly important. From physical keys to electronic access cards, to mobile phones, biometric identification, and other keyless door opening methods, intelligent door control systems have undergone continuous technological innovation. Modern intelligent door control systems integrate technologies such as Internet of Things and cloud computing, enabling remote control, real-time monitoring, visitor management, and other functions, greatly improving convenience and security. With the in-depth application of artificial intelligence, intelligent door control systems will implement face recognition, posture recognition, and other higher-level security authentication, further enhancing security.

[0003] Currently, Chinese patent application No. PCT / CN2016 / 080960 discloses a control method and device for an intelligent door lock and an intelligent door control system, which includes establishing a wireless connection with a mobile terminal (30), receiving an unlock instruction sent by the mobile terminal (30), and determining whether to open the intelligent door lock based on the unlock instruction. This control method and device for the intelligent door lock and the intelligent door control system solve the problem of poor user experience when opening the door. However, traditional intelligent doorbells or door control systems are easily disturbed by complex background noise such as wind noise, traffic, rain noise, and neighbor conversations in outdoor environments. This results in poor signal quality received by the speech recognition (ASR) engine, a significant decrease in recognition accuracy, and users often needing to speak loudly and repeat instructions multiple times, resulting in a poor interactive experience. The speech module and vision module of existing door control systems are isolated, and the lack of situational interaction results in the system's response being preset and rigid, unable to make truly intelligent judgments and responses. The processing flow of traditional intelligent systems is linear and fragile: audio acquisition -> speech-to-text (ASR) -> text understanding (NLU) -> decision. In this chain, any error in any link will result in the failure of the final result. In particular, if the ASR link is incorrect, the subsequent NLU link will receive meaningless input, and the entire interactive chain will be interrupted. The interaction between the traditional system and the user is one-way voice broadcasting. Visitors may need to search around, especially at night or in unfamiliar environments, resulting in low communication efficiency. SUMMARY

[0004] The technical problem addressed by this invention is that the voice and vision modules of existing gating systems are isolated, lacking contextual interaction. This results in only pre-defined and rigid responses, unable to make truly intelligent judgments and responses. The processing flow of traditional intelligent systems is linear and fragile: audio acquisition -> speech-to-text (ASR) -> text understanding (NLU) -> decision-making. An error in any link of this chain will lead to the failure of the final result. Especially in the ASR stage, if a recognition error occurs, the subsequent NLU stage will receive meaningless input, and the entire interaction chain will be interrupted. The interaction between traditional systems and users is one-way voice broadcasting. Visitors may need to search around, especially at night or in unfamiliar environments, resulting in low communication efficiency.

[0005] To address the aforementioned technical problems, this invention provides the following technical solution: an intelligent interactive gate control system, comprising an image acquisition module, a spatial processing module, a voice interaction module, and a control module: The image acquisition module is used to acquire image information in front of the door in real time; The spatial processing module is used to process and analyze the acquired images and human movement trajectories, automatically identify the visitor's identity, and determine their behavior; The voice interaction module is used to generate voice dialogues with visitors; The control module is used to call the voice interaction module to communicate with visitors based on the recognition results of the spatial processing module. It is used for central processing and decision-making, coordinating the work of various modules and executing the final instructions.

[0006] Preferably, the intelligent interactive door control system is used to realize human-computer interaction using an intelligent interactive doorbell, and the workflow of the intelligent interactive doorbell includes: When someone appears in front of the door, the image acquisition module collects image information in real time and sends it to the spatial processing module. The spatial processing module uses a large visual model to analyze images and collects human movement trajectories through motion sensors to identify the identity and behavior of visitors, and sends the results to the control module. Based on the recognition results, the control module calls the corresponding voice interaction logic to control the voice interaction module to engage in dialogue with the visitor. The voice interaction module sends the conversation with the visitor to the control module, which stores the conversation record in the storage module and notifies the owner through the communication module as needed.

[0007] Preferably, the image acquisition module is configured to acquire image information in front of the door in real time by using a high-resolution, high-frame-rate camera, capture facial features of the visitor, and simultaneously monitor spatial position coordinates, behavior actions and posture changes of the visitor in front of the door by using a motion sensor, the behavior actions and posture changes including standing still, holding a package and attempting to turn the handle.

[0008] Preferably, the spatial processing module comprises: The spatial processing module is configured to determine the identity of the visitor, including a courier, a neighbor and a stranger, and identify the body action language and the interactive language of the visitor, the body action language including the behavior actions and the posture changes, and the interactive language including voice recognition utterances of the visitor in front of the door, based on an asymmetric convolutional encoder-decoder structure. The encoder is configured to extract deep visual features of the visitor by combining a multi-layer convolution with an attention mechanism, the deep visual features including a human body edge, a torso region and an action, and fuse the deep visual features with the voice modality in a unified common space, the fusion logic including: The deep visual features are dimensionally reduced by a pooling layer and compressed into a fixed-dimension visual embedding vector, the visual embedding vector including key information of all the body action languages and context information related to vision, and the visual embedding vector is taken as an input of a voice interaction module, the key information and the context information related to vision including a visitor identity, a behavior state and a spatial position relationship changing over time obtained by analyzing target motion and state by the motion sensor; The key information and the context information related to vision obtained by analyzing target motion and state by the motion sensor include: The motion sensor is configured to continuously detect a position, a speed and a body contour of the visitor by a millimeter wave radar, generate three-dimensional point cloud data representing a body shape and a posture in real time, and record a time stamp synchronously; The three-dimensional point cloud data of each visitor is subjected to noise reduction and target tracking processing, three-dimensional coordinates representing key parts of a body of the visitor are extracted from the point cloud data, and continuous coordinates of the same visitor are subjected to Kalman filtering smoothing processing to generate a continuous spatial trajectory; Individualized motion features of the spatial trajectory of each visitor based on the three-dimensional coordinates of the key parts of the body are extracted by a feature extraction algorithm, and the individualized motion features are matched with individualized motion features in a pre-recorded family identity motion database to identify a family identity corresponding to each trajectory; Supplementally identify the identity of the visitor by combining the face recognition technology: real-time collection of facial images through the camera, comparison with the pre-stored face template dataset, updating of the family identity matching, and corresponding matching with the personalized motion features in the pre-recorded family identity motion database to obtain a family identity database, wherein the family identity database includes family identity, family face, and family personalized motion features; Based on the spatial trajectories of each visitor obtained by the motion sensor, the motion parameters of the visitor are calculated, including speed, acceleration, and direction angle. The basic behavior of the target is judged by the real-time basis of the motion parameters, including static, walking, and running. The relative coordinate position relationship between the key points of the visitor's body is analyzed, and the complex action is identified according to the relative coordinate position relationship, which is obtained by the relative coordinate position relationship pre-recorded by the human model. The complex action includes waving hands, bending over, and interaction. The behavior classification library is obtained by classifying the complex action through the machine learning model CNN. The spatial distance of the three-dimensional coordinates of the corresponding body key parts between different visitors is calculated to judge the interaction behavior, which is obtained by comparing the behavior in the behavior classification library. The interaction behavior includes conversation, following, and body conflict. The ID, spatial coordinates, and behavior category of each visitor are sorted by timestamp respectively to generate a dynamic trajectory log, which records the complete state sequence changing over time. The behavior classification library is updated based on the newly collected data, and the family identity database is updated.

[0009] Preferably, the voice interaction module includes a voice collection and enhancement unit, a multi-modal context understanding and embedding unit, a dialogue logic generation and decision unit, and a voice synthesis and visual feedback output unit. The voice collection and enhancement unit is used to capture high-quality and high-intelligibility visitor voice from a complex acoustic environment. The capture process includes: Real-time collection of original omnidirectional sound field audio signals in front of the door through a high-sensitivity microphone array deployed on the door control system, input of the original omnidirectional sound field audio signals into a lightweight door control dilated convolution recurrent network GDCRN for processing, and the processing process includes: Analysis of the time-frequency spectrum of the original omnidirectional sound field audio signals through the internally integrated channel time-frequency attention mechanism CTFA, adaptive allocation of weights to each time-frequency unit including the main voice energy, while suppressing the redundant energy of the noise frequency band, and output of a pure voice stream, wherein the pure voice stream is an audio signal with reduced background noise, prominent human voice, improved signal-to-noise ratio, and intelligibility STOI.

[0010] Preferably, the multi-modal context understanding and embedding unit is used to fuse auditory and visual information, and to have a deep understanding of simple text beyond simple recognition. The fusion process includes: The clean speech stream is input into a Transformer-based speech encoder structure to extract acoustic and linguistic features, generating speech embedding vectors. These visual and speech embedding vectors are then simultaneously input into a fusion network for unified mapping in a common space. This mapping process maps the visual and speech embedding vectors to a unified, high-dimensional semantic common space. The visual embedding vector and the speech embedding vector are normalized, and the normalized visual embedding vector and the speech embedding vector are concatenated to obtain concatenated features. The concatenated features are then processed through the cross attention layer of the Transformer to generate fused features. The fused features are then weighted internally. The cross-modal correlation between visual embedding vectors and speech embedding vectors is enhanced through an attention mechanism. The enhancement process includes: Using the normalized visual embedding vector as the query signal and the normalized speech embedding vector as the key and value, the visual-speech dependency relationship is calculated to obtain the visual-speech relationship vector. The normalized speech embedding vector is used as the query signal, and the normalized visual embedding vector is used as the key and value. The speech-related content features included in the visual information are calculated to obtain the speech-visual relationship vector. The visual-speech relationship vector and the speech-visual relationship vector are added together to obtain the enhanced cross-modal association feature vector; The enhanced cross-modal association feature vectors are mapped to a high-dimensional semantic common space through the output layer of the fusion network; By using the dimensionality reduction visualization technique t-SNE to observe the vector distribution of different semantics in the high-dimensional semantic common space, we can obtain automatically formed classification clusters in the high-dimensional semantic common space. Each classification cluster represents a set of context embedding vectors corresponding to different context types, and the context embedding vectors are used for downstream tasks.

[0011] Preferably, the dialogue logic generation and decision-making unit is used to leverage the powerful generative capabilities of a large language model (LLM) to make intelligent decisions and responses based on a deep understanding of the context. Calling the large language model (LLM) for context-driven processing includes: The context embedding vector is input into the large language model LLM as the core response signal. The large language model LLM dynamically generates the most appropriate text content based on the deep context corresponding to the context embedding vector and the preset owner preferences. Owner preferences include fixed phrases for deliverymen and friendliness towards neighbors.

[0012] Preferably, the speech synthesis and visual feedback output unit is used to convert the decision into an output to the visitor, and the conversion process includes: The text content is fed into a text-to-speech (TTS) engine, which synthesizes a natural and emotional speech for playback, responding to visitors.

[0013] Preferably, the control module includes a coordination unit, a storage unit, and a communication unit: The coordination unit is used for central processing and decision-making, coordinating the work of each module and executing final instructions. The coordination unit includes: Based on the recognition results of the spatial processing module and the dialogue conclusions of the voice interaction module, the system can be linked with the owner's mobile phone and the smart home control system through the communication unit to realize remote video calls, manual remote door opening, and security alarm functions.

[0014] Preferably, the storage unit is used to store the visitor's image information, dialogue records, and preset parameters of the gate control system; The gate control system can be remotely configured and managed by combining local storage and cloud storage. The communication module is used to enable communication between the door control system and external devices, including the owner's mobile phone and smart home system; Information is transmitted through a variety of communication methods, including Wi-Fi, Bluetooth, 4G, and 5G.

[0015] The beneficial effects of this invention are as follows: By deploying a speech enhancement model based on a lightweight GDCRN and attention mechanism at the forefront of signal acquisition, proactive and intelligent noise suppression is achieved. It's not simple filtering, but rather, like the human ear, it "focuses" on the human voice, significantly stripping away noise before it reaches the recognition engine. This fundamentally improves the signal-to-noise ratio of the input signal, providing a high-quality, highly intelligible data source for all subsequent processing steps, ensuring reliable interaction even in harsh acoustic environments. The core of this invention is breaking down the barriers between vision and hearing by fusing visual information (identity, behavior, location) and speech acoustic features at the feature level, generating a unified contextual embedding vector. This means that when the system "understands" a sentence, it simultaneously considers "what it heard" and "what it saw." This allows the system to understand implied meanings, resulting in distinctly different and highly personalized responses. By bypassing the strong dependence on the final text recognition result, it creatively fuses features at the feature level. When making core judgments, the system relies on the contextual embedding vector that integrates acoustic and visual features, rather than a single, potentially erroneous text. This greatly enhances the system's robustness. Even if the text results from ASR are imperfect, as long as the acoustic and visual features match, the system can still make the correct judgment, avoiding the accumulation and amplification of errors in the processing chain. A closed-loop interaction with visual guidance is introduced, where the output is no longer limited to sound but rather a collaborative guidance of "voice + vision." Voice commands are visualized through LED light indicators, screen animations, and other methods, providing users with clear and intuitive operation guidance. Attached Figure Description

[0016] Figure 1 This is a basic flowchart of an intelligent interactive gate control system provided in one embodiment of the present invention. Detailed Implementation

[0017] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0018] Reference Figure 1 As one embodiment of the present invention, an intelligent interactive gate control system is provided, including an image acquisition module, a spatial processing module, a voice interaction module, and a control module: The image acquisition module is used to acquire image information in front of the door in real time; The spatial processing module is used to process and analyze the acquired images and human movement trajectories, automatically identify the visitor's identity, and determine their behavior; The voice interaction module is used to generate voice dialogues with visitors; The control module is used to call the voice interaction module to communicate with visitors based on the recognition results of the spatial processing module. It is used for central processing and decision-making, coordinating the work of various modules and executing the final instructions.

[0019] The intelligent interactive door control system is used to achieve human-computer interaction using an intelligent interactive doorbell. The workflow of the intelligent interactive doorbell includes: When someone appears in front of the door, the image acquisition module collects image information in real time and sends it to the spatial processing module. The spatial processing module uses a large visual model to analyze images and collects human movement trajectories through motion sensors to identify the identity and behavior of visitors, and sends the results to the control module. Based on the recognition results, the control module calls the corresponding voice interaction logic to control the voice interaction module to engage in dialogue with the visitor. For example, if the system identifies the visitor as a courier and the behavior is delivering a package, it can automatically ask the courier for the specific information of the package and inform the courier that the owner is not home at the moment, so that the courier can leave the package at the designated location; if the system identifies the visitor as a neighbor and the behavior is knocking on the door, it can ask the neighbor for the purpose of the visit and respond according to the preset dialogue logic. The voice interaction module sends the conversation with the visitor to the control module, which stores the conversation record in the storage module and notifies the owner through the communication module as needed.

[0020] The owner can remotely view the conversation history and configure and manage the system via external devices such as mobile phones, including modifying preset conversation logic and updating visual and language models.

[0021] The image acquisition module is used to acquire real-time image information in front of the door using a high-resolution, high-frame-rate camera, capture the facial features of visitors, and simultaneously serve as a sensor for the optical tracking system to continuously monitor the spatial coordinates, actions, and posture changes of visitors in front of the door. These actions and posture changes include standing still, holding a package, and attempting to turn the handle, providing accurate dynamic data for subsequent determination of behavioral intentions.

[0022] The space processing module includes: The system uses an asymmetric convolutional encoder-decoder structure to determine the identity of visitors, including deliverymen, neighbors, and strangers. It also identifies the visitors' body language and interactive language, including behavioral actions and posture changes, and interactive language including the visitors' speech at the door. The encoder extracts deep visual features from visitors through multi-layer convolution combined with an attention mechanism. These deep visual features include human body edges, torso regions, and actions, enabling the model to focus on key visual areas such as the face and held objects. These deep visual features are then fused with the speech modality in a unified public space. The fusion logic includes: The deep visual features are reduced in dimensionality through a pooling layer and compressed into a fixed-dimensional visual embedding vector. The visual embedding vector includes key information of all body language and visual contextual information. The visual embedding vector is used as the input of the voice interaction module. The key information and visual contextual information include the visitor's identity, behavioral state and spatial location relationship that change over time by analyzing the target's movement and state through motion sensors. The key information and vision-related contextual information obtained by analyzing the target's motion and state using motion sensors include: The motion sensor continuously detects the visitor's position, speed, and body contour through millimeter-wave radar, generating three-dimensional point cloud data representing body shape and posture in real time, and recording timestamps simultaneously. The 3D point cloud data of each visitor is denoised and target tracking is performed. The 3D coordinates representing key parts of the visitor's body are extracted from the point cloud data. Kalman filtering is performed on the continuous coordinates of the same visitor to generate a continuous spatial trajectory. The personalized motion features of each visitor's spatial trajectory based on the three-dimensional coordinates of key body parts are extracted using feature extraction algorithms. These features, such as gait and posture habits, are then matched with personalized motion features in a pre-recorded family identity motion database to identify the family member identity corresponding to each trajectory. For example, principal component analysis (PCA) algorithm is used for feature extraction. By combining facial recognition technology, the identity of visitors is further identified: facial images are captured in real time by a camera and compared with a pre-stored facial template dataset to update the family member identity matching. The matching is then performed with the personalized motion features in the pre-recorded family member identity motion database to obtain the family member identity database, which includes family member identities, family member faces, and family member personalized motion features. Based on the spatial trajectories of each visitor acquired by motion sensors, motion parameters of the visitors are calculated, including velocity, acceleration, and orientation angle. The basic behavior of the target is determined by the real-world evidence of the motion parameters. The basic behavior includes static, walking, and running. The relative coordinate positional relationship between key points of the visitor's body is analyzed, and complex actions are identified based on the relative coordinate positional relationship. The complex actions are obtained by corresponding to the relative coordinate positional relationship pre-recorded by human models. Complex actions include waving, bending over, and interacting. The complex actions are classified by a machine learning model CNN to obtain a behavior classification library. The spatial distance of the three-dimensional coordinates of corresponding key body parts between different visitors is calculated to determine interactive behavior. The interactive behavior is obtained by comparing with the behaviors in the behavior classification library. Interactive behaviors include talking, following, and physical conflict. The ID, spatial coordinates, and behavior category of each visitor are sorted by timestamp to generate a dynamic trajectory log, which records the complete state sequence that changes over time. For example, a complete state sequence is "10:00:01, Visitor A, Location (10,5,2), Walking status". The behavior classification database is updated based on the newly collected data, and the family member identity database is also updated.

[0023] Set the space size to the working range of the motion sensor.

[0024] The spatial processing module generates human-based embedding vectors, providing rich contextual information for the voice interaction module, enabling collaborative understanding between vision and hearing, and greatly enhancing the intelligence of the interaction.

[0025] The voice interaction module includes a voice acquisition and enhancement unit, a multimodal context understanding and embedding unit, a dialogue logic generation and decision-making unit, and a voice synthesis and visual feedback output unit. The voice interaction module aims to achieve natural, fluent, and highly context-aware interaction with visitors. It abandons the rigid process of traditional "recognition first, understanding later," and achieves an end-to-end closed loop from signal acquisition to intelligent response by integrating visual information with advanced voice processing technology. Its core method is vision-guided multimodal contextual understanding and interaction.

[0026] The speech acquisition and enhancement unit is used to capture high-quality, highly intelligible visitor speech from complex acoustic environments. The capture process includes: The original omnidirectional sound field audio signal in front of the door is acquired in real time by a high-sensitivity microphone array deployed on the gating system. The original omnidirectional sound field audio signal is then input into a lightweight gated dilated convolutional recurrent network (GDCRN) for processing. The processing includes: The original omnidirectional audio signal's time-frequency spectrum is analyzed using an internally integrated Channel Time-Frequency Attention (CTFA) mechanism. Weights are adaptively assigned to each time-frequency unit, including the main speech energy, while redundant energy in noise bands is suppressed, outputting a clean speech stream. This clean speech stream is an audio signal with reduced background noise, prominent human voice, and improved signal-to-noise ratio (SNR) and intelligibility (STOI). The lightweight gated dilated convolutional recurrent network (GDCRN) model is optimized for edge computing devices, featuring low latency and high efficiency. It utilizes gated dilated convolution to expand the receptive field, effectively capturing the contextual information of the speech signal, while modeling non-stationary background noise such as wind noise, traffic noise, and distant human voices.

[0027] The multimodal context understanding and embedding unit is used to fuse auditory and visual information to achieve a deep understanding of simple text that goes beyond simple recognition. The fusion process includes: The clean speech stream is input into a Transformer-based speech encoder structure to extract acoustic and linguistic features, generating speech embedding vectors. These visual and speech embedding vectors are then simultaneously input into a fusion network for unified mapping in a common space. This mapping process maps the visual and speech embedding vectors to a unified, high-dimensional semantic common space. The visual embedding vector and the speech embedding vector are normalized to ensure that the numerical range of the two vectors is consistent. The normalized visual embedding vector and the speech embedding vector are concatenated to obtain the concatenated features. The concatenated features are then processed through the cross attention layer of the Transformer to generate fused features. The fused features are internally weighted, and the features focus more on key semantics. For example, in the scenario of "courier", the model will automatically enhance the association weight between the visual "courier service" feature and the speech keyword "courier". The cross-modal correlation between visual embedding vectors and speech embedding vectors is enhanced through an attention mechanism. The enhancement process includes: The normalized visual embedding vector is used as the query signal, and the normalized speech embedding vector is used as the key and value. The visual-speech dependency relationship is calculated. That is, through the interaction between the visual vector and the speech vector, it is determined which information in the speech is related to the current visual scene, and the relevant information is integrated into the visual vector to obtain the visual-speech relationship vector. The normalized speech embedding vector is used as the query signal, and the normalized visual embedding vector is used as the key and value. The speech-related content features included in the visual information are calculated to obtain the speech-visual relationship vector. The visual-speech relationship vector and the speech-visual relationship vector are added together to obtain the enhanced cross-modal association feature vector; The enhanced cross-modal association feature vector is mapped to a high-dimensional semantic common space through the output layer of the fusion network; for example, a linear transformation layer with ReLU activation is used to transform the 2048-dimensional fusion features into a 1536-dimensional context embedding vector, which contains joint "visual-speech" semantic information.

[0028] By using the t-SNE dimensionality reduction visualization technique, we can observe the vector distribution of different semantics in the high-dimensional semantic public space and obtain the automatically formed classification clusters in the high-dimensional semantic public space. Each classification cluster represents a set of context embedding vectors corresponding to different context types, such as a courier saying hello and a neighbor saying hello. The context embedding vectors are then used for downstream tasks.

[0029] Downstream tasks, for example: Context classification: Determine the visitor's behavioral intention, such as "delivering a package" or "neighborhood interaction"; By using voice queries to find matching visual scenes, or by using visual scenes to retrieve corresponding voice content, the semantic consistency of public spaces can be verified.

[0030] Example: Analysis of the integration process distinguishing between "delivery person" and "neighbor": Visual feature extraction: Extract visual features such as clothing (delivery uniform or casual clothes) and items carried (delivery parcel or empty hands) from the image and generate corresponding visual embedding vectors; Speech feature extraction: Extract the acoustic features of "hello" from speech, and understand the context through a language model, such as "my package has arrived" or "the weather is nice today", to generate speech embedding vectors.

[0031] The cross-modal fusion process includes: Through the cross-attention mechanism, the visual vector will focus on keywords related to "express delivery" in the speech, while the speech vector will focus on features such as "package" in the vision. After mapping in public space, the contextual vector of "the courier says hello" integrates the joint semantics of "courier uniform + package + courier-related vocabulary", while the vector of "the neighbor says hello" integrates the semantics of "casual clothes + empty hands + everyday conversational vocabulary". The two form a clear distance difference in public space.

[0032] Based on the generated context vectors, the system can directly determine the behavior category, such as "express delivery" or "neighborhood greeting", or provide contextualized response basis for intelligent customer service.

[0033] This method, through cross-modal fusion, can more fully understand the joint semantics of "visual-speech" and solve the ambiguity problem that cannot be distinguished by a single modality, such as the semantic differences of the same speech text in different visual scenarios.

[0034] The dialogue logic generation and decision-making unit utilizes the powerful generative capabilities of the large-scale language model (LLM) to make intelligent decisions and responses based on a deep understanding of the context. Context-driven decision-making by invoking the large-scale language model (LLM) includes: The context embedding vector is input into the large language model LLM as the core response signal. The large language model LLM dynamically generates the most appropriate text content based on the deep context corresponding to the context embedding vector and the preset owner preferences. Owner preferences include fixed phrases for deliverymen and friendliness towards neighbors.

[0035] The speech synthesis and visual feedback output unit is used to convert decisions into outputs to visitors, and the conversion process includes: The text content is fed into a text-to-speech (TTS) engine, which synthesizes a natural and emotional speech for playback, responding to visitors.

[0036] For example, when the voice announces "Please place your package on the shelf to the right of the door," an LED indicator on the door control system will light up and point in the direction of the shelf, providing visitors with clear visual guidance.

[0037] This dual guidance of "voice + vision" greatly improves the efficiency of interaction and user experience, especially in noisy or poorly lit environments.

[0038] Through these four closely linked steps, the voice interaction module constructs a complete closed loop from perception, enhancement, fusion understanding to intelligent feedback, achieving true intelligent interaction.

[0039] The control module includes a coordination unit, a storage unit, and a communication unit: The coordination unit is used for central processing and decision-making, coordinating the work of each module and executing final instructions. The coordination unit includes: Based on the recognition results of the spatial processing module and the dialogue conclusions of the voice interaction module, the system can be linked with the owner's mobile phone and the smart home control system through the communication unit to realize remote video calls, manual remote door opening, and security alarm functions.

[0040] For example, once the visual module recognizes the visitor as the "owner" or the visitor's identity is confirmed by voice interaction, the module will issue a door-opening command.

[0041] The storage unit is used to store the visitor's image information, dialogue records, and preset parameters of the gate control system; By combining local storage and cloud storage, the access control system can be remotely set up and managed, allowing the owner to easily view historical records at any time. The communication module is used to enable communication between the door control system and external devices, including the owner's mobile phone and smart home system; Information is transmitted through a variety of communication methods, including Wi-Fi, Bluetooth, 4G, and 5G.

[0042] The specific implementation steps of the gate control system are as follows: If (away mode): Automatically triggers AI doorbell answering; Else (Away mode not set): Whether to intervene and respond depends on the user's "AI intelligent response based on scenario" switch; Whether to intervene and respond depends on the user's "AI intelligent response based on scenario" intervention time; The doorbell call will automatically interrupt the response process. What is the expected timeframe for AI to make a judgment / provide feedback in each scenario? The AI-involved aspects include: Visual judgment = preliminary assessment of a person's purpose for visiting (less than 3 seconds); Voice judgment = the intention behind a person's spoken words; Visual judgment = whether there is a dangerous intent (less than 3 seconds); Model dialogue = judging the semantics of the visitor's speech + responding to the visitor (natural and realistic feeling); The door control system detected a human figure approaching from a distance at the doorway (3 seconds); If (doorbell rings && no response within 5 seconds): If the visitor brings items: Print("Hello, this is your smart home assistant aosu. Your owner is currently unavailable to come out and greet you. Are you here to deliver a package?"). If (delivering packages / delivering takeout): Print("Hello, you can leave the package at the location designated by the owner"); Case (for packages / delivery orders) OF A: Place it directly in front of the door; If (gated visible): Print(“Thank you, have a nice day”); Else: Print(“Sorry, please place the package directly in front of the door”); If (gated visible): Print(“Thank you, have a nice day”); Else: Print(“I have recorded the delivery information and notified the owner that you may leave now. Have a pleasant day”); B: Come at another time; C: Put it at the neighbor's house; D: Place in a parcel locker; OTHERWISE: Leave a message for the homeowner; Else: Print(“Hello, I am the smart butler aosu. The owner is currently unable to come out to greet visitors. Please state your identity and the purpose of your visit in one sentence”). If (it is determined that the visitor's answer corresponds to a pre-defined visitor scenario): Print("Okay, I will inform the master of your visit. You may leave now. Have a pleasant time."); Else: Print("I didn't understand the purpose of your visit, but I have informed the host of your visit. You may leave now. Have a pleasant day."); Else: Visitor Greetings; Print(“Hello, I am the smart butler aosu. The owner is currently unable to come out to greet visitors. Please state your identity and the purpose of your visit in one sentence”). If (identified as a visitor scenario): Print(“Okay, I will inform the master of your visit. You may leave now. Have a pleasant time.”); Check if you have left after 8 seconds; If (not left): Abnormal hovering around P0; Print(“Your stay has exceeded the permitted time. Your behavior has been recorded. Please leave immediately. The police are being connected!”); warn; Call the owner; Else: Return 0; Else: Print("I didn't understand the purpose of your visit, but I have informed the host of your visit. You may leave now. Have a pleasant day."); Else (Face detection recording): If (a family member's face is recognized that is on record): Print(“welcome”); Else: If (identification of armed personnel, weapon-wielding individuals, sabotage, or package theft): alarm; Call the owner; Else: Print(“Who are you? I will inform my master”); If (identified as a visitor scenario): Print(“Okay, I will inform the master of your visit.”); Else: Print("I didn't understand the purpose of your visit, but I have informed the host of your visit. You may leave now."); Check if you have left after 8 seconds; If (not left): Abnormal hovering around P0; Print(“Your stay has exceeded the permitted time. Your behavior has been recorded. Please leave immediately. The police are being connected!”); warn; Call the owner; Else: Return 0.

[0043] This invention achieves proactive and intelligent noise suppression by deploying a speech enhancement model based on a lightweight GDCRN and attention mechanism at the forefront of signal acquisition. It's not simply filtering, but rather "focusing" on the human voice like the human ear, significantly stripping away noise before it reaches the recognition engine. This fundamentally improves the signal-to-noise ratio of the input signal, providing a high-quality, highly intelligible data source for all subsequent processing, ensuring reliable interaction even in harsh acoustic environments. The core of this invention breaks down the barriers between vision and hearing by fusing visual information (identity, behavior, location) and speech acoustic features at the feature level, generating a unified contextual embedding vector. This means that when the system "understands" a sentence, it simultaneously considers "what it heard" and "what it saw." This allows the system to understand implied meanings and thus make distinctly different, highly personalized responses. By bypassing a strong dependence on the final text recognition result, it creatively fuses features at the feature level. When making core judgments, the system relies on the contextual embedding vector that integrates acoustic and visual features, rather than a single, potentially erroneous text. This greatly enhances the system's robustness. Even if the text results from ASR are imperfect, as long as the acoustic and visual features match, the system can still make the correct judgment, avoiding the accumulation and amplification of errors in the processing chain. A closed-loop interaction with visual guidance is introduced, where the output is no longer limited to sound but rather a collaborative guidance of "voice + vision." Voice commands are visualized through LED light indicators, screen animations, and other methods, providing users with clear and intuitive operation guidance.

[0044] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0045] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. An intelligent interactive gate control system, characterized in that, It includes an image acquisition module, a spatial processing module, a voice interaction module, and a control module: The image acquisition module is used to acquire image information in front of the door in real time; The spatial processing module is used to process and analyze the acquired images and human movement trajectories, automatically identify the visitor's identity, and determine their behavior; The voice interaction module is used to generate voice dialogues with visitors; The control module is used to call the voice interaction module to communicate with visitors based on the recognition results of the spatial processing module. It is used for central processing and decision-making, coordinating the work of various modules and executing the final instructions.

2. The intelligent interactive gate control system as described in claim 1, characterized in that, The intelligent interactive door control system is used to achieve human-computer interaction using an intelligent interactive doorbell. The workflow of the intelligent interactive doorbell includes: When someone appears in front of the door, the image acquisition module collects image information in real time and sends it to the spatial processing module. The spatial processing module uses a large visual model to analyze images and collects human movement trajectories through motion sensors to identify the identity and behavior of visitors, and sends the results to the control module. Based on the recognition results, the control module calls the corresponding voice interaction logic to control the voice interaction module to engage in dialogue with the visitor. The voice interaction module sends the conversation with the visitor to the control module, which stores the conversation record in the storage module and notifies the owner through the communication module as needed.

3. The intelligent interactive gate control system as described in claim 2, characterized in that, The image acquisition module is used to acquire real-time image information in front of the door using a high-resolution, high-frame-rate camera, capture the facial features of visitors, and simultaneously use motion sensors to continuously monitor the spatial coordinates, actions, and posture changes of visitors in front of the door. The actions and posture changes include standing still, holding a package, and attempting to turn the handle.

4. The intelligent interactive gate control system as described in claim 3, characterized in that, The space processing module includes: The system uses an asymmetric convolutional encoder-decoder structure to determine the identity of visitors, including deliverymen, neighbors, and strangers. It also identifies the visitors' body language and interactive language, including behavioral actions and posture changes, and interactive language including the visitors' speech at the door. The encoder extracts deep visual features from visitors through multi-layer convolution combined with an attention mechanism. These deep visual features include human body edges, torso regions, and movements. These deep visual features are then fused with the speech modality in a unified public space. The fusion logic includes: The deep visual features are reduced in dimensionality through a pooling layer and compressed into a fixed-dimensional visual embedding vector. The visual embedding vector includes key information of all body language and visual contextual information. The visual embedding vector is used as the input of the voice interaction module. The key information and visual contextual information include the visitor's identity, behavioral state and spatial location relationship that change over time by analyzing the target's movement and state through motion sensors. The key information and vision-related contextual information obtained by analyzing the target's motion and state using motion sensors include: The motion sensor continuously detects the visitor's position, speed, and body contour through millimeter-wave radar, generating three-dimensional point cloud data representing body shape and posture in real time, and recording timestamps simultaneously. The 3D point cloud data of each visitor is denoised and target tracking is performed. The 3D coordinates representing key parts of the visitor's body are extracted from the point cloud data. Kalman filtering is performed on the continuous coordinates of the same visitor to generate a continuous spatial trajectory. The personalized motion features of each visitor's spatial trajectory based on the three-dimensional coordinates of key body parts are extracted by feature extraction algorithms and matched with the personalized motion features in the pre-entered family identity motion database to identify the family member identity corresponding to each trajectory. By combining facial recognition technology, the identity of visitors is further identified: facial images are captured in real time by a camera and compared with a pre-stored facial template dataset to update the family member identity matching. The matching is then performed with the personalized motion features in the pre-recorded family member identity motion database to obtain the family member identity database, which includes family member identities, family member faces, and family member personalized motion features. Based on the spatial trajectories of each visitor acquired by motion sensors, motion parameters of the visitors are calculated, including velocity, acceleration, and orientation angle. The basic behavior of the target is determined by the real-world evidence of the motion parameters. The basic behavior includes static, walking, and running. The relative coordinate positional relationship between key points of the visitor's body is analyzed, and complex actions are identified based on the relative coordinate positional relationship. The complex actions are obtained by corresponding to the relative coordinate positional relationship pre-recorded by human models. Complex actions include waving, bending over, and interacting. The complex actions are classified by a machine learning model CNN to obtain a behavior classification library. The spatial distance of the three-dimensional coordinates of corresponding key body parts between different visitors is calculated to determine interactive behavior. The interactive behavior is obtained by comparing with the behaviors in the behavior classification library. Interactive behaviors include talking, following, and physical conflict. Each visitor's ID, spatial coordinates, and behavior category are sorted by timestamp to generate a dynamic trajectory log, recording the complete state sequence that changes over time. The behavior classification database is updated based on the newly collected data, and the family member identity database is also updated.

5. The intelligent interactive gate control system as described in claim 4, characterized in that, The voice interaction module includes a voice acquisition and enhancement unit, a multimodal context understanding and embedding unit, a dialogue logic generation and decision-making unit, and a voice synthesis and visual feedback output unit. The speech acquisition and enhancement unit is used to capture high-quality, highly intelligible visitor speech from complex acoustic environments. The capture process includes: The original omnidirectional sound field audio signal in front of the door is acquired in real time by a high-sensitivity microphone array deployed on the gating system. The original omnidirectional sound field audio signal is then input into a lightweight gated dilated convolutional recurrent network (GDCRN) for processing. The processing includes: The time-frequency attention mechanism (CTFA) is integrated internally to analyze the time spectrum of the original omnidirectional sound field audio signal, adaptively assign weights to each time-frequency unit including the main speech energy, and suppress redundant energy in the noise frequency band to output a clean speech stream. The clean speech stream is an audio signal with reduced background noise, prominent human voice, and improved signal-to-noise ratio and intelligibility (STOI).

6. The intelligent interactive gate control system as described in claim 5, characterized in that, The multimodal context understanding and embedding unit is used to fuse auditory and visual information to achieve a deep understanding of simple text that goes beyond simple recognition. The fusion process includes: The clean speech stream is input into a Transformer-based speech encoder structure to extract acoustic and linguistic features, generating speech embedding vectors. These visual and speech embedding vectors are then simultaneously input into a fusion network for unified mapping in a common space. This mapping process maps the visual and speech embedding vectors to a unified, high-dimensional semantic common space. The visual embedding vector and the speech embedding vector are normalized, and the normalized visual embedding vector and the speech embedding vector are concatenated to obtain concatenated features. The concatenated features are then processed through the cross attention layer of the Transformer to generate fused features. The fused features are then weighted internally. The cross-modal correlation between visual embedding vectors and speech embedding vectors is enhanced through an attention mechanism. The enhancement process includes: Using the normalized visual embedding vector as the query signal and the normalized speech embedding vector as the key and value, the visual-speech dependency relationship is calculated to obtain the visual-speech relationship vector. The normalized speech embedding vector is used as the query signal, and the normalized visual embedding vector is used as the key and value. The speech-related content features included in the visual information are calculated to obtain the speech-visual relationship vector. The visual-speech relationship vector and the speech-visual relationship vector are added together to obtain the enhanced cross-modal association feature vector; The enhanced cross-modal association feature vectors are mapped to a high-dimensional semantic common space through the output layer of the fusion network; By using the dimensionality reduction visualization technique t-SNE to observe the vector distribution of different semantics in the high-dimensional semantic common space, we can obtain automatically formed classification clusters in the high-dimensional semantic common space. Each classification cluster represents a set of context embedding vectors corresponding to different context types, and the context embedding vectors are used for downstream tasks.

7. The intelligent interactive gate control system as described in claim 6, characterized in that, The dialogue logic generation and decision-making unit utilizes the powerful generative capabilities of the large-scale language model (LLM) to make intelligent decisions and responses based on a deep understanding of the context. Context-driven decision-making by invoking the large-scale language model (LLM) includes: The context embedding vector is input into the large language model LLM as the core response signal. The large language model LLM dynamically generates the most appropriate text content based on the deep context corresponding to the context embedding vector and the preset owner preferences. Owner preferences include fixed phrases for deliverymen and friendliness towards neighbors.

8. The intelligent interactive gate control system as described in claim 7, characterized in that, The speech synthesis and visual feedback output unit is used to convert decisions into outputs to visitors, and the conversion process includes: The text content is fed into a text-to-speech (TTS) engine, which synthesizes a natural and emotional speech for playback, responding to visitors.

9. The intelligent interactive gate control system as described in claim 8, characterized in that, The control module includes a coordination unit, a storage unit, and a communication unit: The coordination unit is used for central processing and decision-making, coordinating the work of each module and executing final instructions. The coordination unit includes: Based on the recognition results of the spatial processing module and the dialogue conclusions of the voice interaction module, the system can be linked with the owner's mobile phone and smart home control system through the communication unit to realize remote video calls, manual remote door opening, and security alarm functions.

10. The intelligent interactive gate control system as described in claim 9, characterized in that, The storage unit is used to store the visitor's image information, dialogue records, and preset parameters of the gate control system; The gate control system can be remotely configured and managed by combining local storage and cloud storage. The communication module is used to enable communication between the door control system and external devices, including the owner's mobile phone and smart home system; Information is transmitted through a variety of communication methods, including Wi-Fi, Bluetooth, 4G, and 5G.

Images