Intelligent interactive door control system

Abstract

The application discloses an intelligent interactive door control system, comprising an image acquisition device, a voice acquisition device, a millimeter wave radar sensor, a voice playing device, a visual indication device, a memory and a processor. The processor is configured to determine the identity category of a visitor according to the image collected by the image acquisition device and the motion data collected by the millimeter wave radar sensor, perform voice enhancement processing and voice changing processing on the voice of the visitor collected by the voice acquisition device, fuse the visual information and the voice information of the visitor to generate a first vector, generate voice dialogue content according to the first vector and preset user preferences, and perform video processing on the monitoring video collected by the image acquisition device to generate video summary information. Through front-end voice enhancement, voice changing processing, cross-modal context fusion and video event intelligent summary, the application realizes intelligent interaction, privacy protection and efficient information acquisition of the door control system in multiple scenes.

Description

Technical Field

[0001] This application relates to the field of intelligent security technology, specifically to a door control system that integrates voice interaction and visual understanding. Background Technology

[0002] With the continuous advancement of technology and the widespread application of the Internet of Things (IoT), intelligent security has become a crucial component of modern social life and enterprise management. Traditional security systems are limited by simple physical protection and limited monitoring capabilities, while modern intelligent security systems are making great strides towards integration, networking, and intelligence, placing greater emphasis on user experience, security efficiency, and real-time response. As a key facility at building entrances and exits, the improvement of the intelligence and interactive experience of doors is becoming increasingly important. From physical keys to electronic access cards, and then to keyless opening methods such as mobile phones and biometrics, intelligent door control systems have undergone continuous technological innovation. Modern intelligent door control systems integrate technologies such as the IoT and cloud computing, enabling remote control, real-time monitoring, and visitor management, greatly improving convenience and security. Furthermore, with the deepening application of artificial intelligence, intelligent door control systems will achieve higher levels of security authentication such as facial recognition and posture recognition, further enhancing security.

[0003] Regarding the input quality of voice interaction, traditional smart doorbells or door control systems are highly susceptible to interference from complex background noise such as wind noise, traffic, rain, and neighborly conversations in outdoor environments. This results in poor signal quality received by the voice recognition engine, significantly reducing recognition accuracy. Users often need to shout and repeat commands multiple times, leading to a very poor interactive experience. Furthermore, when vulnerable groups such as women living alone, the elderly, and children use doorbells for intercom, their actual voices directly reveal their gender and age, providing criminals with information to determine the situation of people inside, posing a serious security risk. Existing door control systems lack effective means of protecting voice privacy.

[0004] Regarding the viewing experience of surveillance videos, even if existing gate control systems record a full day's worth of surveillance video and interactive events, users still need to manually drag and drop repeatedly on a lengthy timeline to find the video content they are interested in, making the viewing process extremely costly. Key content in surveillance videos is mixed in with a large number of ineffective still images and images of people not present, making it easy for users to be obscured by a deluge of irrelevant content. Users struggle to quickly locate and grasp the key dynamics of the entire day, and there is no intelligent categorization for review based on personnel type (such as delivery personnel or strangers). Summary of the Invention

[0005] The technical problem this application addresses is that existing gating systems use isolated voice and vision devices, lacking contextual interaction. This results in pre-defined, rigid responses that fail to provide truly intelligent judgments and responses. Traditional intelligent systems operate on a linear and fragile process: audio acquisition -> speech-to-text (ASR) -> text understanding (NLU) -> decision-making. An error in any link of this chain leads to failure of the final result. Particularly in the ASR stage, a recognition error results in meaningless input to the subsequent NLU stage, disrupting the entire interaction chain. Furthermore, traditional systems interact with users through one-way voice broadcasts. Visitors may need to search for the system, especially at night or in unfamiliar environments, leading to inefficient communication.

[0006] To address the aforementioned technical problems, this application provides the following technical solution: an intelligent interactive door control system, comprising an image acquisition device, a processor, a voice interaction device, and the processor itself: the image acquisition device is used to acquire image information in front of the door in real time; the processor is used to process and analyze the acquired images and human movement trajectories, automatically identify the visitor's identity, and determine their behavior; the voice interaction device is used to generate a voice dialogue with the visitor; the processor is used to invoke the voice interaction device to engage in dialogue with the visitor based on the recognition results of the processor, for central processing and decision-making, coordinating the work of each device, and executing the final instructions.

[0007] The beneficial effects of this application are as follows: First, by combining the image acquisition device and the millimeter-wave radar sensor for collaborative perception, and integrating visual features and three-dimensional spatial trajectory information, it achieves all-weather, highly robust identification of visitors. Even in complex doorway environments such as drastic changes in lighting and visual obstruction, it can still accurately distinguish between deliverymen, neighbors, strangers and family members, and output differentiated control signals accordingly, overcoming the shortcomings of single-modal perception being susceptible to environmental interference. Second, by deploying a speech enhancement model based on a lightweight gated dilated convolutional recurrent network and a channel time-frequency attention mechanism at the forefront of signal acquisition, active and intelligent noise suppression is achieved. This is not simple filtering, but rather "focusing" on the human voice like the human ear, significantly stripping away noise before it reaches the recognition engine. This fundamentally improves the signal-to-noise ratio of the input signal, ensuring reliable interaction even in harsh acoustic environments such as wind noise and traffic. Simultaneously, through a three-stage voice-changing process involving timbre conversion, distortion correction, and audio enhancement, the output speech is converted to the target timbre while maintaining high naturalness, effectively concealing the user's true gender and age information, significantly improving the personal safety of vulnerable groups such as women living alone, the elderly, and children using the gating system. Third, the core of this application is breaking down the barriers between vision and hearing. By fusing visual information (identity, behavior, spatial location) and speech acoustic features through bidirectional cross-attention at the feature level, a unified first vector is generated. This means that when the system "understands" a sentence, it simultaneously combines "what it heard" and "what it saw," enabling it to comprehend implied meaning and make highly personalized responses. By bypassing the strong dependence on the final text recognition result, the system creatively integrates features at the feature level. It makes core judgments based on a first vector that combines acoustic and visual features, rather than a single, potentially erroneous text. Even if the text result from speech recognition is imperfect, as long as the acoustic and visual features match, the system can still make a correct judgment, avoiding the accumulation and amplification of errors in the processing chain. Fourth, by inputting the first vector into a large language model and combining it with preset user preferences, the system can generate natural, fluent, and context-appropriate dialogue content and control the output of the voice playback device, achieving a leap from "fixed mapping response" to "generative intelligent dialogue." Simultaneously, the system introduces collaborative guidance from visual indicators (such as LED lights indicating direction and screen animations displaying operations), forming a closed-loop interaction with dual "voice + vision" guidance. Fifth, by performing intelligent video processing on surveillance videos, it automatically removes static and unattended scenes to extract dynamic video clips, filters target content based on preset event tags, classifies personnel by demographic attributes and generates exclusive video collections corresponding to different demographic categories, and automatically generates video text summaries and representative cover images. This allows users to efficiently review the entire day's surveillance content in a short time without having to repeatedly drag and search on a lengthy monitoring timeline, significantly reducing review costs and improving information acquisition efficiency. Attached Figure Description

[0008] Figure 1 This is a basic flowchart of an intelligent interactive gate control system provided in one embodiment of this application.

[0009] Figure 2 This is another technical flow diagram of an intelligent interactive gate control system provided in one embodiment of this application.

[0010] Figure 3 This is a schematic diagram of the voice changing process of an intelligent interactive gate control system provided in one embodiment of this application.

[0011] Figure 4 This is a flowchart illustrating the video processing and event summarization of an intelligent interactive gating system provided in one embodiment of this application. Detailed Implementation

[0012] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the specific embodiments of this application will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0013] Reference Figure 1 As one embodiment of this application, an intelligent interactive gating system is provided, including an image acquisition device 10, a spatial processing device, a voice interaction device, and a control device: Image acquisition device 10 is used to acquire image information in front of the door in real time; The spatial processing device is used to process and analyze the acquired images and human movement trajectories, automatically identify the visitor's identity, and determine the behavior; optionally, the spatial processing device includes a processor 70 and a motion sensor, which may be a millimeter-wave radar sensor 30.

[0014] The voice interaction device is used to generate voice dialogue with visitors; optionally, the voice interaction device includes a voice acquisition device 20 and a voice playback device 40.

[0015] The control unit is used to invoke the voice interaction device to engage in dialogue with the visitor based on the recognition results of the spatial processing device. It serves as a central processing and decision-making unit, coordinating the operation of various devices and executing final instructions. Optionally, the control unit also includes a processor 70, which can be a separate processor.

[0016] The intelligent interactive door control system is used to achieve human-computer interaction using an intelligent interactive doorbell. The workflow of the intelligent interactive doorbell includes: When someone appears in front of the door, the image acquisition device 10 acquires image information in real time and sends it to the spatial processing device. The processor 70 of the spatial processing device uses a large visual model to analyze images and collects human movement trajectories through motion sensors to identify the identity and behavior of visitors and send the results to the control device. Based on the recognition results, the control device invokes the corresponding voice interaction logic to control the voice interaction device to engage in dialogue with the visitor.

[0017] For example, if the system identifies the visitor as a courier and the behavior is delivering a package, it can automatically ask the courier for the specific information of the package and tell the courier to put the package at the designated location; if the system identifies the visitor as a neighbor and the behavior is knocking on the door, it can ask the neighbor for the purpose of the visit and respond according to the preset dialogue logic. The voice interaction device sends the conversation with the visitor to the control device, which stores the conversation record in the storage device and notifies the owner through the communication device as needed.

[0018] The owner can remotely view the conversation history and configure and manage the system via external devices such as mobile phones, including modifying preset conversation logic and updating visual and language models.

[0019] Reference Figure 2 The image acquisition device 10 is used to acquire image information in front of the door in real time using a high-resolution, high-frame-rate camera, capture the facial features of visitors, and at the same time serve as a sensor of the optical tracking system to continuously monitor the spatial coordinates, behavior and posture changes of visitors in front of the door. The behavior and posture changes include standing still, holding a package and trying to turn the handle, providing accurate dynamic data for subsequent behavior intention judgment.

[0020] The spatial processing device is used to: determine the identity of visitors based on an asymmetric convolutional encoder-decoder structure, including the identities of deliverymen, neighbors, and strangers, and to identify the visitors' body language and interactive language, including behavioral actions and posture changes, and interactive language including the visitors' speech recognition speech at the door. The encoder extracts deep visual features from visitors through multi-layer convolution combined with an attention mechanism. These deep visual features include human body edges, torso regions, and actions, enabling the model to focus on key visual areas such as the face and held objects. These deep visual features are then fused with the speech modality in a unified public space. The fusion logic includes: The deep visual features are reduced in dimensionality through a pooling layer and compressed into a second vector of fixed dimension. The second vector includes key information of all body language and visual contextual information. The second vector is used as the input of the voice interaction device. The key information and visual contextual information include the visitor's identity, behavioral state and spatial location relationship that change over time, obtained by analyzing the target's movement and state through motion sensors. The key information and vision-related contextual information obtained by analyzing the target's motion and state using motion sensors include: The motion sensor continuously detects the visitor's position, speed, and body contour through millimeter-wave radar, generating three-dimensional point cloud data representing body shape and posture in real time, and recording timestamps simultaneously. The 3D point cloud data of each visitor is denoised and target tracking is performed. The 3D coordinates representing key parts of the visitor's body are extracted from the point cloud data. Kalman filtering is performed on the continuous coordinates of the same visitor to generate a continuous spatial trajectory. The personalized motion features of each visitor's spatial trajectory based on the three-dimensional coordinates of key body parts are extracted using feature extraction algorithms. These features, such as gait and posture habits, are then matched with personalized motion features in a pre-recorded family identity motion database to identify the family member identity corresponding to each trajectory. For example, principal component analysis (PCA) algorithm is used for feature extraction. By combining facial recognition technology, the identity of visitors is further identified: facial images are captured in real time by a camera and compared with a pre-stored facial template dataset to update the family member identity matching. The matching is then performed with the personalized motion features in the pre-recorded family member identity motion database to obtain the family member identity database, which includes family member identities, family member faces, and family member personalized motion features. Based on the spatial trajectories of each visitor acquired by motion sensors, motion parameters of the visitors are calculated, including velocity, acceleration, and orientation angle. The basic behavior of the target is determined by the real-world evidence of the motion parameters. The basic behavior includes static, walking, and running. The relative coordinate positional relationship between key points of the visitor's body is analyzed, and complex actions are identified based on the relative coordinate positional relationship. The complex actions are obtained by corresponding to the relative coordinate positional relationship pre-recorded by human models. Complex actions include waving, bending over, and interacting. The complex actions are classified by a machine learning model CNN to obtain a behavior classification library. The spatial distance of the three-dimensional coordinates of corresponding key body parts between different visitors is calculated to determine interactive behavior. The interactive behavior is obtained by comparing with the behaviors in the behavior classification library. Interactive behaviors include talking, following, and physical conflict. The ID, spatial coordinates, and behavior category of each visitor are sorted by timestamp to generate a dynamic trajectory log, which records the complete state sequence that changes over time. For example, a complete state sequence is "10:00:01, Visitor A, Location (10,5,2), Walking status". The behavior classification database is updated based on the newly collected data, and the family member identity database is also updated.

[0021] Set the space size to the working range of the motion sensor.

[0022] The spatial processing device generates human-based embedding vectors, providing rich contextual information for voice interaction devices, enabling collaborative understanding of vision and hearing, and greatly enhancing the intelligence of the interaction.

[0023] The voice interaction device includes a voice acquisition and enhancement unit, a multimodal context understanding and embedding unit, a dialogue logic generation and decision-making unit, and a voice synthesis and visual feedback output unit. The voice interaction device aims to achieve natural, fluent, and highly context-aware interaction with visitors. It abandons the rigid process of traditional "recognition first, understanding later," and achieves an end-to-end closed loop from signal acquisition to intelligent response by integrating visual information with advanced voice processing technology. Its core method is based on visually guided multimodal contextual understanding and interaction.

[0024] The speech acquisition and enhancement unit is used to capture high-quality, highly intelligible visitor speech from complex acoustic environments. The capture process includes: The original omnidirectional sound field audio signal in front of the door is acquired in real time by a high-sensitivity microphone array deployed on the gating system. The original omnidirectional sound field audio signal is then input into a lightweight gated dilated convolutional recurrent network (GDCRN) for processing. The processing includes: The original omnidirectional audio signal's time-frequency spectrum is analyzed using an internally integrated Channel Time-Frequency Attention (CTFA) mechanism. Weights are adaptively assigned to each time-frequency unit, including the main speech energy, while redundant energy in noise bands is suppressed, outputting a clean speech stream. This clean speech stream is an audio signal with reduced background noise, prominent human voice, and improved signal-to-noise ratio (SNR) and intelligibility (STOI). The lightweight gated dilated convolutional recurrent network (GDCRN) model is optimized for edge computing devices, featuring low latency and high efficiency. It utilizes gated dilated convolution to expand the receptive field, effectively capturing the contextual information of the speech signal, while modeling non-stationary background noise such as wind noise, traffic noise, and distant human voices.

[0025] The multimodal context understanding and embedding unit is used to fuse auditory and visual information to achieve a deep understanding of simple text that goes beyond simple recognition. The fusion process includes: The clean speech stream is input into a Transformer-based speech encoder structure to extract acoustic and linguistic features, generating a third vector. The second and third vectors are then simultaneously input into a fusion network for unified mapping in a common space. This mapping process maps the second and third vectors to a unified, high-dimensional semantic common space. The second and third vectors are normalized to ensure that their numerical ranges are consistent. The normalized second and third vectors are then concatenated to obtain concatenated features. These concatenated features are then processed through the cross-attention layer of the Transformer to generate fused features. The fused features are then internally weighted, with the features focusing more on key semantics. For example, in the scenario of "courier", the model will automatically enhance the association weight between the visual feature of "courier service" and the speech keyword "courier". The cross-modal correlation between the second and third vector mappings is enhanced through an attention mechanism. The enhancement process includes: The normalized second vector is used as the query signal, and the normalized third vector is used as the key and value. The visual dependence on speech is calculated. That is, through the interaction between the visual vector and the speech vector, it is determined which information in the speech is related to the current visual scene, and the relevant information is integrated into the visual vector to obtain the visual-speech relationship vector. The normalized third vector is used as the query signal, and the normalized second vector is used as the key and value. The speech-related content features included in the visual information are calculated to obtain the speech-visual relationship vector. The visual-speech relationship vector and the speech-visual relationship vector are added together to obtain the enhanced cross-modal association feature vector; The enhanced cross-modal association feature vector is mapped to a high-dimensional semantic common space through the output layer of the fusion network; for example, a linear transformation layer with ReLU activation is used to transform the 2048-dimensional fusion features into a 1536-dimensional first vector, which contains joint "visual-speech" semantic information.

[0026] By observing the vector distribution of different semantics in the high-dimensional semantic public space using the dimensionality reduction visualization technique t-SNE, we can obtain automatically formed classification clusters in the high-dimensional semantic public space. Each classification cluster represents a set of first vectors corresponding to different context types, such as a courier saying hello and a neighbor saying hello. The first vectors are then used for downstream tasks.

[0027] Downstream tasks, for example: Context classification: Determine the visitor's behavioral intention, such as "delivering a package" or "neighborhood interaction"; By using voice queries to find matching visual scenes, or by using visual scenes to retrieve corresponding voice content, the semantic consistency of public spaces can be verified.

[0028] Example: Analysis of the integration process distinguishing between "delivery person" and "neighbor": Visual feature extraction: Extract visual features such as clothing (delivery uniform or casual clothes) and items carried (delivery parcel or empty hands) from the image and generate the corresponding second vector; Speech feature extraction: Extract the acoustic features of "hello" from speech, and understand the context through a language model, such as "my package has arrived" or "the weather is nice today", to generate a third vector.

[0029] The fusion process includes: Through the cross-attention mechanism, the visual vector will focus on keywords related to "express delivery" in the speech, while the speech vector will focus on features such as "package" in the vision. After mapping in public space, the contextual vector of "the courier says hello" integrates the joint semantics of "courier uniform + package + courier-related vocabulary", while the vector of "the neighbor says hello" integrates the semantics of "casual clothes + empty hands + everyday conversational vocabulary". The two form a clear distance difference in public space.

[0030] Based on the generated context vectors, the system can directly determine the behavior category, such as "express delivery" or "neighborhood greeting", or provide contextualized response basis for intelligent customer service.

[0031] This method, through fusion, can more fully understand the joint semantics of "visual-speech" and resolve ambiguities that cannot be distinguished by a single modality, such as the semantic differences of the same speech text in different visual scenarios.

[0032] The dialogue logic generation and decision-making unit utilizes the powerful generative capabilities of the large-scale language model (LLM) to make intelligent decisions and responses based on a deep understanding of the context. Context-driven decision-making by invoking the large-scale language model (LLM) includes: The first vector is input as the core response signal into the large language model LLM. The large language model LLM generates the most appropriate text content based on the deep context corresponding to the first vector and the preset owner preferences. Owner preferences include fixed phrases for deliverymen and friendliness towards neighbors.

[0033] The speech synthesis and visual feedback output unit is used to convert decisions into outputs to visitors, and the conversion process includes: The text content is fed into a text-to-speech (TTS) engine, which synthesizes a natural and emotional speech for playback, responding to visitors.

[0034] For example, when the voice announces "Please place your package on the shelf to the right of the door," an LED indicator on the door control system will light up and point in the direction of the shelf, providing visitors with clear visual guidance.

[0035] In yet another embodiment of this application, such as Figure 3 As shown, the voice interaction device also integrates voice changing processing functionality. However, related technologies suffer from drawbacks such as low naturalness and a noticeable mechanical feel when changing a user's voice. Therefore, this application provides a voice changing method that can improve the naturalness of the voice change and reduce the mechanical feel.

[0036] Specifically, the voice changing processing method includes: receiving intercom voice; and performing voice changing processing S220 on the intercom voice according to the mapping relationship between the user's voice timbre and the target voice timbre to obtain the changed voice. The target voice timbre can be a different voice than the user's, such as a deep male voice or a clown's voice. To achieve the voice changing effect S220, user voice timbre features can be extracted from the intercom voice, input into a voice changing control model to obtain voice changing control parameters, and then the intercom voice can be processed using voice changing control parameters S220. The voice changing mapping model can be trained based on user voice samples and target voice samples, supporting non-linear mapping effects.

[0037] After the timbre conversion process S220, the converted speech is processed for distortion correction S230 based on the audio distortion data from the audio acquisition device to obtain the corrected speech. Specifically, this includes: dividing the converted speech into speech frames and converting them into speech spectra; determining the fundamental frequency of each speech frame and querying the corresponding audio distortion data in a data table based on the fundamental frequency; and using the audio distortion data to perform distortion correction on the speech spectrum S230 to obtain the corrected speech spectrum, which is then reconstructed into the corrected speech. The data table stores the audio distortion data corresponding to each fundamental frequency from the audio acquisition device; this data can be pre-determined using user speech samples collected by the audio acquisition device.

[0038] After distortion correction S230, based on the spectral difference between the corrected speech and the sample speech of the target timbre, the frequency band to be enhanced in the corrected speech is determined, and audio enhancement S240 is performed on the frequency band to be enhanced to obtain the output speech. Specifically, the average harmonic envelope determined based on the sample speech is obtained, the difference spectrum between the spectrum of the corrected speech and the average harmonic envelope is determined, the preset frequency band is divided into continuous frequency bands according to the preset frequency range, and the continuous frequency band with the largest integral value of the difference spectrum is set as the frequency band to be enhanced; the spectrum of the corrected speech is boosted within the frequency band to be enhanced, the positions of each harmonic peak in the boosted spectrum are phase aligned, and the aligned spectrum is reconstructed as the output speech.

[0039] Through the three-stage processing flow of timbre conversion (S220), distortion correction (S230), and audio enhancement (S240), the distortion of the output speech can be effectively reduced, while the fullness and recognizability of the output speech can be improved, thereby enhancing the quality of voice changing. Especially when women living alone, the elderly, or children use doorbell intercoms, voice changing can naturally conceal the user's true gender and age information, improving user safety.

[0040] This dual guidance of "voice + vision" greatly improves the efficiency of interaction and user experience, especially in noisy or poorly lit environments.

[0041] Through these four closely linked steps, the voice interaction device constructs a complete closed loop from perception, enhancement, fusion understanding to intelligent feedback, achieving true intelligent interaction.

[0042] The control device includes a coordination unit, a storage unit, and a communication unit: The coordination unit is used for central processing and decision-making, coordinating the work of various devices and executing final instructions. The coordination unit includes: Based on the recognition results of the spatial processing device and the dialogue conclusions of the voice interaction device, the communication unit can be linked with the owner's mobile phone and the smart home control system to realize remote video calls, manual remote door opening, and security alarm functions.

[0043] For example, once the visual device recognizes the "owner" or the visitor's identity is confirmed by voice interaction, the device will issue a door-opening command.

[0044] The storage unit is used to store the visitor's image information, dialogue records, and preset parameters of the gate control system; By combining local storage and cloud storage, the access control system can be remotely set up and managed, allowing the owner to easily view historical records at any time. The communication device is used to enable communication between the door control system and external devices, including the owner's mobile phone and smart home system; Information is transmitted through a variety of communication methods, including Wi-Fi, Bluetooth, 4G, and 5G.

[0045] In another embodiment of this application, such as Figure 4 As shown, the gate control system also integrates video processing and event summarization functions. Currently, in the design of video surveillance systems for reviewing 24-hour monitoring videos, users can only manually drag and drop repeatedly on the timeline of the monitoring video to find video content of interest or focus. This is costly, and the key content of the monitoring video is easily obscured by a large amount of irrelevant content, making it difficult for users to quickly locate and grasp the key points of the monitoring video.

[0046] To address this, this application provides a video processing method that reduces the cost of user playback. The method includes: identifying dynamic video segments containing human activity from surveillance video. Specifically, each frame of the surveillance video is identified to remove invalid images; based on the surveillance video after removing invalid images, dynamic video segments containing human activity are identified; wherein, invalid images are static images that do not change within a preset time period, or video images that do not contain human activity. A preset recognition model based on artificial intelligence is used to analyze the surveillance video frame by frame; video frames that do not change within a preset time period and video frames that do not contain human activity are identified as invalid images and removed; after removing invalid images, the remaining temporally continuous images are aggregated into dynamic video segments containing human activity.

[0047] Subsequently, target video segments are selected from each dynamic video clip based on preset event tags. Preset event tags are labels used to describe the types of events that the user is interested in, such as "someone enters," "abnormal loitering," "package delivery," and "vehicle enters." The dynamic video clips are then further filtered using preset event tags to remove segments that are irrelevant to the events that the user is interested in, ultimately resulting in one or more target video clips after information denoising.

[0048] Then, crowd attribute classification (S530) is performed on the people appearing in the target video segment to obtain at least one crowd attribute category corresponding to the target video segment. Specifically, this includes: extracting biometric features from each person appearing in the target video segment; if there is a first person whose biometric features match preset features, then the first person is determined to belong to the target crowd attribute category; if there is a second person whose biometric features do not match preset features, then the multimodal contextual features of the second person are extracted from the target video segment, and the crowd attribute category to which the second person belongs is determined based on the multimodal contextual features. The multimodal contextual features include at least two of the following: clothing features, behavioral features, location features, and object interaction features.

[0049] Next, based on each target video segment involving the same population attribute category, a dedicated video collection corresponding to that same population attribute category is generated. The same target video segment can belong to at least one dedicated video collection. By recording references or indexes pointing to the corresponding target video segments in the dedicated video collections, duplicate storage of video data for the same target video segment can be avoided.

[0050] Finally, corresponding video description information is generated for the dedicated video collection, including video text summaries and / or video covers. For the video text summaries, a model with visual understanding capabilities can be used to perform visual understanding of each video segment in the dedicated video collection, identifying semantic content such as people, behaviors, and events, and generating a generalized video text summary accordingly. For the video covers, each frame can be evaluated based on preset evaluation dimensions (such as image quality, subject sharpness, and event representativeness), and the frame with the highest image score can be selected as the video cover.

[0051] Through the above video processing, long-duration surveillance videos can be automatically converted into short video clips that can be quickly viewed, as well as exclusive video collections categorized by audience attributes (such as deliverymen, strangers, etc.). This enables efficient, accurate, and personalized playback of video highlights, reduces the cost of playback for users, allows users to quickly grasp the key points of surveillance videos, and improves the efficiency of information acquisition from surveillance videos.

[0052] Furthermore, the video processing function described above in this application also supports executing corresponding response operations based on user instructions obtained from the user terminal: when the user instruction is a first instruction, the monitoring video is extracted from the original video based on the custom capture range in the first instruction; when the user instruction is a second instruction, the corresponding video processing flow is triggered based on the custom trigger event in the second instruction; when the user instruction is a third instruction, a corresponding exclusive video collection is generated based on the specified population attribute category in the third instruction.

[0053] The specific implementation steps of the gate control system are as follows: If (away mode): Automatically triggers AI doorbell answering; Else (Away mode not set): Whether to intervene and respond depends on the user's "AI intelligent response based on scenario" switch; Whether to intervene and respond depends on the user's "AI intelligent response based on scenario" intervention time; The doorbell call will automatically interrupt the response process. What is the expected timeframe for AI to make a judgment / provide feedback in each scenario? The AI-involved aspects include: Visual judgment = preliminary assessment of a person's purpose for visiting (less than 3 seconds); Voice judgment = the intention behind a person's spoken words; Visual judgment = whether there is a dangerous intent (less than 3 seconds); Model dialogue = judging the semantics of the visitor's speech + responding to the visitor (natural and realistic feeling); The door control system detected a human figure approaching from a distance at the doorway (3 seconds); If (doorbell rings && no response within 5 seconds): If the visitor brings items: Print("Hello, this is your smart home assistant aosu. Your owner is currently unavailable to come out and greet you. Are you here to deliver a package?"). If (delivering packages / delivering food): Print("Hello, you can leave the package at the location designated by the owner"); Case (for packages / delivery orders) OF A: Place it directly in front of the door; If (gated visible): Print(“Thank you, have a nice day”); Else: Print(“Sorry, please place the package directly in front of the door”); If (gated visible): Print(“Thank you, have a nice day”); Else: Print(“I have recorded the delivery information and notified the owner that you may leave now. Have a pleasant day”); B: Come at another time; C: Put it at the neighbor's house; D: Place in a parcel locker; OTHERWISE: Leave a message for the homeowner; Else: Print(“Hello, I am the smart butler aosu. The owner is currently unable to come out to greet visitors. Please state your identity and the purpose of your visit in one sentence”). If (it is determined that the visitor's answer corresponds to a pre-defined visitor scenario): Print("Okay, I will inform the master of your visit. You may leave now. Have a pleasant time."); Else: Print("I didn't understand the purpose of your visit, but I have informed the host of your visit. You may leave now. Have a pleasant day."); Else: Visitor Greetings; Print(“Hello, I am the smart butler aosu. The owner is currently unable to come out to greet visitors. Please state your identity and the purpose of your visit in one sentence”). If (identified as a visitor scenario): Print(“Okay, I will inform the master of your visit. You may leave now. Have a pleasant time.”); Check if you have left after 8 seconds; If (not left): Abnormal hovering around P0; Print(“Your stay has exceeded the permitted time. Your behavior has been recorded. Please leave immediately. The police are being connected!”); warn; Call the owner; Else: Return 0; Else: Print("I didn't quite catch the purpose of your visit, but I've informed the host of your arrival. You may leave now. Have a pleasant day."); Else (Face detection recording): If (a family member's face is recognized that is on record): Print(“welcome”); Else: If (identification of armed personnel, weapon-wielding individuals, sabotage, or package theft): alarm; Call the owner; Else: Print(“Who are you? I will inform my master”); If (identified as a visitor scenario): Print(“Okay, I will inform the master of your visit.”); Else: Print("I didn't understand the purpose of your visit, but I have informed the host of your visit. You may leave now."); Check if you have left after 8 seconds; If (not left): Abnormal hovering around P0; Print(“Your stay has exceeded the permitted time. Your behavior has been recorded. Please leave immediately. The police are being connected!”); warn; Call the owner; Else: Return 0.

[0054] This application has the following beneficial effects: First, by combining the image acquisition device and the millimeter-wave radar sensor for collaborative perception, and integrating visual features and three-dimensional spatial trajectory information, it achieves all-weather, highly robust identification of visitors. Even in complex doorway environments such as drastic changes in lighting and visual obstruction, it can still accurately distinguish between deliverymen, neighbors, strangers, and family members, and output differentiated control signals accordingly, overcoming the shortcomings of single-modal perception being susceptible to environmental interference. Second, by deploying a speech enhancement model based on a lightweight gated dilated convolutional recurrent network and a channel time-frequency attention mechanism at the forefront of signal acquisition, active and intelligent noise suppression is achieved. This is not simple filtering, but rather "focusing" on the human voice like the human ear, significantly stripping away noise before it reaches the recognition engine. This fundamentally improves the signal-to-noise ratio of the input signal, ensuring reliable interaction even in harsh acoustic environments such as wind noise and traffic. Simultaneously, through a three-stage voice-changing process involving timbre conversion, distortion correction, and audio enhancement, the output speech is converted to the target timbre while maintaining high naturalness, effectively concealing the user's true gender and age information, significantly improving the personal safety of vulnerable groups such as women living alone, the elderly, and children using the gating system. Third, the core of this application is breaking down the barriers between vision and hearing. By fusing visual information (identity, behavior, spatial location) and speech acoustic features through bidirectional cross-attention at the feature level, a unified first vector is generated. This means that when the system "understands" a sentence, it simultaneously combines "what it heard" and "what it saw," enabling it to comprehend implied meaning and make highly personalized responses. By bypassing the strong dependence on the final text recognition result, the system creatively integrates features at the feature level. It makes core judgments based on a first vector that combines acoustic and visual features, rather than a single, potentially erroneous text. Even if the text result from speech recognition is imperfect, as long as the acoustic and visual features match, the system can still make a correct judgment, avoiding the accumulation and amplification of errors in the processing chain. Fourth, by inputting the first vector into a large language model and combining it with preset user preferences, the system can generate natural, fluent, and context-appropriate dialogue content and control the output of the voice playback device, achieving a leap from "fixed mapping response" to "generative intelligent dialogue." Simultaneously, the system introduces collaborative guidance from visual indicators (such as LED lights indicating direction and screen animations displaying operations), forming a closed-loop interaction with dual "voice + vision" guidance. Fifth, by performing intelligent video processing on surveillance videos, it automatically removes static and unattended scenes to extract dynamic video clips, filters target content based on preset event tags, classifies personnel by demographic attributes and generates exclusive video collections corresponding to different demographic categories, and automatically generates video text summaries and representative cover images. This allows users to efficiently review the entire day's surveillance content in a short time without having to repeatedly drag and search on a lengthy monitoring timeline, significantly reducing review costs and improving information acquisition efficiency.

[0055] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0056] It should be noted that the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit it. Although this application has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of this application without departing from the spirit and scope of the technical solutions of this application, and all such modifications and substitutions should be covered within the scope of the claims of this application.

Claims

1. A smart interactive gate control system, characterized in that, It includes an image acquisition device, a voice acquisition device, a millimeter-wave radar sensor, a voice playback device, a visual indication device, a memory, and a processor; the memory stores a computer program; when the processor executes the computer program, it is used for: Based on the images acquired by the image acquisition device and the motion data acquired by the millimeter-wave radar sensor, the identity category of the visitor is determined and a control signal is output. The system receives the original audio signal output by the voice acquisition device through the voice enhancement interface, and outputs the enhanced voice signal and voice-changing processing. Visual and verbal information from visitors are fused to generate a first vector; Based on the first vector and preset user preferences, generate voice dialogue content with the visitor, and control the voice playback device to output the corresponding voice. The visual guidance device is controlled to provide visual guidance according to the identity category; video processing is performed on the surveillance video acquired by the image acquisition device to generate video summary information.

2. The intelligent interactive gate control system as described in claim 1, characterized in that, The processor is used to implement the speech enhancement processing in the following manner: Noise suppression is achieved by using a gated dilated convolutional recurrent network and a channel time-frequency attention mechanism to suppress noise in the original audio signal and output a clean speech stream. Based on the mapping relationship between the user's timbre and the target timbre, the pure speech stream is sequentially subjected to timbre conversion processing, distortion correction processing, and audio enhancement processing to obtain the output speech; Based on the mapping relationship between the user's timbre and the target timbre extracted from the image and motion data, the pure speech stream is sequentially processed by timbre conversion, distortion correction and audio enhancement to obtain the output speech. The second vector extracted from the image and motion data is fused with the third vector of the output speech through a cross-attention mechanism to generate the first vector; The first vector is input into a large language model, and the voice dialogue content is generated by combining it with the preset user preferences.

3. The intelligent interactive gate control system as described in claim 1, characterized in that, The processor is used to implement the voice-changing process in the following manner: Based on the mapping relationship between the user's voice and the target voice, the visitor's voice is processed to convert the voice and obtain the converted voice. Based on the audio distortion data from the speech acquisition device, the converted speech is subjected to distortion correction processing to obtain corrected speech; Based on the spectral difference between the corrected speech and the sample speech of the target timbre, the frequency band to be enhanced in the corrected speech is determined, and audio enhancement processing is performed on the frequency band to be enhanced to obtain the output speech.

4. The intelligent interactive gate control system as described in claim 3, characterized in that, The processor is used to implement the distortion correction process in the following manner: The converted speech is divided into speech frames, and the speech frames are converted into speech spectra; Determine the fundamental frequency of the speech frame, and query the corresponding audio distortion data in the pre-stored data table based on the fundamental frequency; The pre-stored data table contains audio distortion data corresponding to each base frequency of the voice acquisition device. The audio distortion data is pre-determined using user voice samples collected by the voice acquisition device. The audio distortion data is used to correct the distortion of the speech spectrum to obtain the corrected speech spectrum; the corrected speech spectrum is then reconstructed into the corrected speech.

5. The intelligent interactive gate control system as described in claim 1, characterized in that, The processor is used to determine the visitor's identity category in the following ways: The identity of the visitor is determined by an asymmetric convolutional encoder-decoder structure. The visitor's identity categories include multiple preset identity categories such as deliveryman, neighbor, and stranger. Deep visual features of visitors are extracted by combining multi-layer convolution with attention mechanism, and the deep visual features are compressed into a second vector through pooling layer.

6. The intelligent interactive gate control system as described in claim 5, characterized in that, The processor is also used for: The three-dimensional point cloud data detected by the millimeter-wave radar sensor is denoised and the target is tracked. The three-dimensional coordinates of key parts are extracted from the denoised three-dimensional point cloud data. The key parts are preset parts of the visitor's body. Continuous spatial trajectories are generated through Kalman filtering smoothing. Personalized motion features are extracted based on the spatial trajectory and matched with the family member identity motion database pre-recorded in the memory to identify family members.

7. The intelligent interactive gate control system as described in claim 6, characterized in that, The processor is used to perform the video processing in the following manner: Identify dynamic video segments from the surveillance video that show human activity; Target video segments are selected from each of the aforementioned dynamic video segments based on preset event tags; The people appearing in the target video segment are classified according to their crowd attributes to obtain at least one crowd attribute category corresponding to the target video segment; Based on each of the target video segments involving the same group attribute category, generate a dedicated video collection corresponding to the same group attribute category; Generate corresponding video description information for the exclusive video collection, including video text summaries and / or video covers.

8. The intelligent interactive gate control system as described in claim 7, characterized in that, The processor is used to determine the dynamic video segment in the following manner: Each frame of the surveillance video is identified, and still images that do not change within a preset time and images that do not contain people are removed. Based on the remaining images after removal, dynamic video segments containing human activity were identified.

9. The intelligent interactive gate control system as described in claim 7, characterized in that, The processor is used to classify the population attributes in the following ways: Biometric features are extracted from each person appearing in the target video segment; If a first person exists whose biometrics match the preset characteristics, then the first person is determined to belong to the target population attribute category. If a second person exists whose biometric features do not match the preset features, then the multimodal context features of the second person are extracted from the target video segment, and the population attribute category to which the second person belongs is determined based on the multimodal context features; wherein, the multimodal context features include at least two of clothing features, behavioral features, location features and item interaction features, and the same target video segment belongs to at least one of the exclusive video collections.

10. The intelligent interactive gate control system as described in claim 7, characterized in that, The processor is also used for: Based on all the target video segments, generate overall video description information for the surveillance video; Execute corresponding response operations based on user instructions obtained from the user terminal. The user instructions include custom capture range instructions, custom trigger event instructions, and / or instructions specifying population attribute categories.

Images