Intelligent door control method based on multi-modal information, intelligent door and program product

Abstract

The invention provides an intelligent door control method based on multi-modal information, an intelligent door and a program product. The intelligent door control method comprises the steps that when it is detected that a person approaches outside a door, image information, audio information and environment parameters outside the door are collected; performing face recognition and personnel behavior analysis according to the image information, and determining a face confidence coefficient and a behavior confidence coefficient between the detected personnel outside the door and the registered user; performing voiceprint detection according to the audio information, and determining a first voiceprint confidence degree between the detected person and the registered user; determining a confidence reliability score according to the environmental parameters; determining a total confidence score of the detected person and the registered user according to the face confidence, the behavior confidence, the first voiceprint confidence and the confidence reliability score; and controlling the intelligent door to be opened or kept closed according to the total confidence score.

Description

Technical Field

[0001] The present disclosure relates to an intelligent door control method based on multimodal information, an intelligent door and a program product. Background Art

[0002] With the widespread adoption of smart home technology, users are placing higher demands on the security and interactivity of smart door systems. Existing smart door systems mostly rely on a single recognition method, such as facial recognition, voiceprint recognition, or password verification. However, single recognition modes are vulnerable to spoofing attacks, such as the use of 3D facial masks or fingerprint films. Furthermore, these systems have poor adaptability in complex environments. For example, the accuracy of facial or voiceprint recognition can significantly decrease under extreme lighting or high noise conditions. Summary of the Invention

[0003] The present disclosure provides a smart door control method based on multimodal information, a smart door, and a program product.

[0004] According to one aspect of the present disclosure, a smart door control method based on multimodal information is provided, including: when a person approaching outside the door is detected, collecting image information, audio information and environmental parameters outside the door; performing face recognition and person behavior analysis based on the image information, and determining the face confidence and behavior confidence between the detected person outside the door and a registered user; performing voiceprint detection based on the audio information, and determining the first voiceprint confidence between the detected person and the registered user; determining a confidence reliability score based on the environmental parameters; determining a total confidence score between the detected person and the registered user based on the face confidence, behavior confidence, first voiceprint confidence and the confidence reliability score; and controlling the smart door to open or remain closed based on the total confidence score.

[0005] According to one technical solution, by combining multimodal information such as facial recognition, voiceprint detection, and behavioral analysis, multi-factor cross-validation is implemented, effectively preventing counterfeit attacks such as 3D facial masks or fingerprint films. Compared to single recognition methods, the false positive rate is significantly reduced, improving the security and reliability of the smart door system. Furthermore, a confidence reliability score is determined based on environmental parameters and incorporated into the calculation of the total confidence score, allowing the system to adapt to different environmental conditions, improve recognition accuracy and stability, and ensure the normal operation of the smart door in various environments.

[0006] According to the method of at least one embodiment of the present disclosure, the total confidence score of the detected person and the registered user is determined based on the facial confidence, the behavioral confidence, the first voiceprint confidence and the confidence reliability score, including: obtaining the weight values ​​corresponding to the facial confidence, the behavioral confidence, the first voiceprint confidence and the confidence reliability score; performing weighted sum calculation based on the facial confidence, the behavioral confidence, the first voiceprint confidence, the confidence reliability score and the weight values ​​to obtain the total confidence score of the detected person corresponding to the registered user.

[0007] According to the technical solution of this embodiment, the confidence levels of multiple biometric and behavioral characteristics can be integrated and calculated, which increases the difficulty of forgery and deception and improves the security of the smart door system.

[0008] According to the method of at least one embodiment of the present disclosure, the smart door is controlled to open or remain closed according to the total confidence score, including: comparing the maximum value of the total confidence score of the detected person corresponding to each registered user with the score threshold; when the maximum value of the total confidence score is greater than or equal to the score threshold, controlling the smart door to open; when the maximum value of the total confidence score is less than the score threshold, controlling the smart door to remain closed.

[0009] According to the technical solution of this embodiment, by setting a score threshold, the door is allowed to be opened only when the total confidence score is high enough, effectively preventing illegal intrusion and improving the security of the smart door.

[0010] According to the method of at least one embodiment of the present disclosure, before weighted and calculated according to the facial confidence, the behavioral confidence, the first voiceprint confidence, the confidence reliability score and the weight value, it also includes: adjusting the weight values ​​corresponding to the facial confidence, the behavioral confidence and / or the first voiceprint confidence according to the environmental parameters, and the environmental parameters include at least one of light intensity, sound decibels and the number of people outside the door.

[0011] According to the technical solution of this embodiment, subsequent weighted sum calculations are performed based on the adjusted weight values, which can reduce misidentification caused by environmental interference.

[0012] According to the method of at least one embodiment of the present disclosure, the method further includes: when a person approaching the door is detected, collecting audio information inside the door; performing semantic recognition and voiceprint detection based on the audio information inside the door, and determining the semantic information and the second voiceprint confidence between the detected person inside the door and the registered user; when the voice information is the target semantics and the second voiceprint confidence is greater than the voiceprint confidence threshold, controlling the smart door to open.

[0013] According to the technical solution of this embodiment, through the dual verification of semantic recognition and voiceprint detection, it is possible to correctly identify whether the user has gone out, reducing the possibility of misidentification.

[0014] According to the method of at least one embodiment of the present disclosure, it also includes: during the closing process of the smart door, detecting whether there is an object in the door gap area; when there is an object in the door gap area or the motor drive current of the smart door is greater than or equal to a current threshold, controlling the smart door to stop closing and opening the smart door in reverse.

[0015] According to the technical solution of this embodiment, the accuracy of anti-pinch detection is improved through the dual detection mechanism of real-time detection of the door gap area and the motor drive current, ensuring that obstacles can be reliably detected in various situations, effectively avoiding pinching injuries, and improving the safety of smart doors.

[0016] According to the method of at least one embodiment of the present disclosure, it also includes: performing personnel behavior detection based on the image information, and determining the behavior detection result of the detected person outside the door, wherein the personnel behavior detection result is used to characterize whether the detected person outside the door has abnormal behavior, and the abnormal behavior includes at least one of wandering outside the door and tailgating behavior; in the case that the detected person outside the door has abnormal behavior, the opening signal of the smart door is suppressed.

[0017] According to the technical solution of this embodiment, abnormal behavior outside the door can be effectively identified, potential intruders or suspicious persons can be discovered in time, illegal intrusions can be prevented, and the safety of the living environment can be improved.

[0018] According to another aspect of the present disclosure, a smart door is provided, which integrates: a multimodal sensor group, including a binocular camera, an audio array and an infrared sensor; a main control unit, which is electrically connected to the multimodal sensor group, and the main control unit is used to execute the method as described in any one of the present disclosures according to the detection information of the multimodal sensor group; and an actuator, which is electrically connected to the main control unit, and the actuator is used to drive the smart door to open or close according to the control instructions of the main control unit.

[0019] According to at least one embodiment of the smart door of the present disclosure, it further includes: an interactive module, which is electrically connected to the main control unit, and the interactive module includes at least one of an inner door touch screen, an outer door display screen, and a status indication unit.

[0020] According to at least one embodiment of the present disclosure, the smart door further includes: a power management module, the power management module including a main power link, a backup power supply and a power switching circuit, wherein the main power link is connected to the mains power, and the power switching circuit is used to switch to the backup power supply for power supply when the mains power is interrupted.

[0021] According to another aspect of the present disclosure, an electronic device is provided, comprising: a memory storing execution instructions; and a processor executing the execution instructions stored in the memory, so that the processor executes the method of any embodiment of the present disclosure.

[0022] According to another aspect of the present disclosure, a readable storage medium is provided, wherein the readable storage medium stores execution instructions, and when the execution instructions are executed by a processor, the execution instructions are used to implement the method of any embodiment of the present disclosure.

[0023] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the method of any embodiment of the present disclosure is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings illustrate exemplary embodiments of the present disclosure and together with the description serve to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification.

[0025] Figure 1 A flow chart of a smart door control method based on multimodal information according to an embodiment of the present disclosure is shown.

[0026] Figure 2 FIG. 1 is a flow chart of step S150 according to an embodiment of the present disclosure.

[0027] Figure 3 A flow chart of step S160 according to an embodiment of the present disclosure is shown.

[0028] Figure 4 A schematic diagram of a process flow of exit control included in a smart door control method based on multimodal information according to an embodiment of the present disclosure is shown.

[0029] Figure 5 A schematic flow chart of anti-pinch control included in the intelligent door control method based on multimodal information according to one embodiment of the present disclosure is shown.

[0030] Figure 6 A schematic diagram of a process for detecting abnormal behavior outside a door, which is also included in a smart door control method based on multimodal information according to an embodiment of the present disclosure, is shown.

[0031] Figure 7 A schematic diagram of a smart door according to an embodiment of the present disclosure is shown.

[0032] Figure 8 A schematic diagram of a smart door framework according to another embodiment of the present disclosure is shown.

[0033] Figure 9 A schematic diagram of a smart door framework according to another embodiment of the present disclosure is shown.

[0034] Figure 10 A schematic structural block diagram of an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0035] The present disclosure is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific examples described herein are intended only to illustrate the relevant content and are not intended to limit the present disclosure. It should also be noted that, for ease of description, only the portions relevant to the present disclosure are shown in the accompanying drawings.

[0036] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in the present disclosure can be combined with each other. The technical solution of the present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0037] With the rapid development of smart home technology, users and homeowners have placed higher expectations on the intelligence, interactivity, security, and reliability of their entrance doors. However, most door control systems currently on the market rely on a single recognition method (such as facial recognition, fingerprint recognition, or password verification), exposing a series of problems that need to be addressed.

[0038] First, the single nature of the recognition method poses a significant security risk. A single recognition mode is highly susceptible to counterfeit attacks. For example, a 3D facial mask could potentially breach the facial recognition system's defenses, and a fingerprint film could deceive the sensor. However, the system generally lacks multi-factor cross-verification mechanisms, making it unable to effectively mitigate such risks.

[0039] Secondly, the access control system performs poorly in terms of environmental adaptability. The recognition module's performance degrades significantly in extreme lighting conditions, such as backlight or low-light environments, and in scenes with severe acoustic interference (such as high ambient noise), making it difficult to maintain stable and reliable recognition results.

[0040] To this end, the present disclosure proposes the following technical solution, which combines multimodal information such as facial recognition, voiceprint detection, and behavioral analysis to achieve multi-factor cross-validation. This solution effectively prevents counterfeit attacks, such as 3D facial masks or fingerprint films, and significantly reduces the false recognition rate compared to single recognition methods, thereby improving the security and reliability of the smart door system. Furthermore, a confidence reliability score is determined based on environmental parameters and incorporated into the calculation of the total confidence score, thereby adapting to different environmental conditions, improving recognition accuracy and stability, and ensuring the normal operation of the smart door in various environments.

[0041] Figure 1 FIG. 1 shows a flow chart of a smart door control method based on multimodal information according to an embodiment of the present disclosure. Figure 1 As shown, the method may include steps S110 to S160. The method may be executed by a main control unit configured on the smart door, or by an electronic device such as a smart phone or a tablet computer.

[0042] In step S110, when it is detected that a person is approaching outside the door, image information, audio information and environmental parameters outside the door are collected.

[0043] In this embodiment, an infrared sensor may be installed on the outside of the smart door. When the infrared sensor detects that a person has entered the detection range (which can be pre-set, for example, within 3 meters of the door), the subsequent information collection process is triggered.

[0044] An image acquisition device (such as a binocular camera) can be installed on the smart door. When a person approaches outside the door, the image acquisition device can be activated to collect image information of the area outside the door.

[0045] In one example, the image acquisition device may capture images of the area outside the door at regular intervals to obtain image information of the outside. In another example, the image acquisition device may capture a real-time video stream of the outside and then extract frames from the video stream at regular intervals to obtain the corresponding image information. It should be noted that those skilled in the art may select between the two acquisition methods based on actual implementation needs, and these are not particularly limited.

[0046] In one example, when the smart door is powered by a mains connection, the aforementioned method of first capturing a video stream and then extracting frames can be used to obtain image information outside the door. When the smart door is powered by an independent power source or a backup power source, the aforementioned image capture method can be used to obtain image information, thereby avoiding excessive power consumption due to video capture and achieving rational energy utilization.

[0047] Smart doors can also be equipped with an embedded audio array (consisting of one or more microphones) to collect audio signals from outside. The audio array uses beamforming technology to focus on the direction of the voice of the person outside, reducing background noise interference. When a person approaches, the audio array activates to capture audio information from outside.

[0048] Smart doors can also be equipped with environmental sensors (such as light sensors, noise sensors, etc.). When a person approaches the door, the environmental sensors can be activated to measure the environmental parameters of the current environment (such as light intensity, noise level, etc.).

[0049] Please continue to refer to Figure 1 In step S120, face recognition and personnel behavior analysis are performed based on the image information to determine the face confidence and behavior confidence between the detected person outside the door and the registered user.

[0050] Among them, face recognition can be a process of analyzing facial features in an image and comparing them with facial features of registered users to determine the similarity between the two facial features.

[0051] The face confidence level can be a quantitative score of the degree of matching between the facial features of the detected person outside the door and the registered user.

[0052] Personnel behavior analysis can be a process of using image information to identify and analyze the behavior patterns of people outside the door (such as standing posture, walking posture, door pressing, etc.), extracting corresponding behavioral features, and comparing them with the behavioral features of registered users to determine the similarity between the two behavioral features.

[0053] The behavioral confidence level can be a quantitative score of the degree of matching between the behavioral characteristics of the detected person outside the door and the registered user.

[0054] In this embodiment, the collected image information can be preprocessed (e.g., grayscaling, denoising, contrast enhancement, etc.) to improve image quality. Next, a face detection algorithm can be used to detect the facial region in the image and perform face alignment. Feature vectors (128D / 512D vectors for ArcFace and FaceNet) are then extracted from the facial region. Similarity calculations (e.g., cosine similarity calculations) are performed between the extracted facial region feature vectors and the facial feature vectors of registered users. The resulting similarity value is the face confidence level.

[0055] It should be noted that there may be multiple registered users, and a similarity calculation may be performed on the facial feature vector of each registered user to obtain the facial confidence level corresponding to the detected person and each registered user.

[0056] Based on the preprocessed image information, the detected person's behavior can be identified to determine the degree of match between the detected person's behavioral characteristics and those of registered users. Specifically, in addition to image information, the infrared sensor installed on the smart door can also be used to collect behavioral characteristics of the detected person, including standing posture, walking posture, door pressing, and other movements. A temporal convolutional neural network (TCN), long short-term memory network (LSTM), or Transformer model is then used to extract a behavioral feature sequence. The detected person's behavioral feature sequence is compared with the behavioral feature sequence of the registered user, and a match score is generated using methods such as trajectory entropy and action similarity, resulting in a corresponding behavioral confidence level.

[0057] Similarly, when there are multiple registered users, the behavioral confidence between the detected person and each registered user can be obtained through the above calculation.

[0058] Please continue to refer to Figure 1 In step S130, voiceprint detection is performed based on the audio information to determine the first voiceprint confidence between the detected person and the registered user.

[0059] The voiceprint detection may be a process of analyzing the voiceprint features in the audio information and comparing them with the voiceprint features of the registered user to determine the similarity between the two voiceprint features.

[0060] The first voiceprint confidence level may be a quantitative score of the degree of matching between the voiceprint features of the person outside the door and the registered user, reflecting the credibility of the identity recognition.

[0061] In this implementation, the audio information collected by the audio array can be preprocessed to extract MFCCs (Mel-Frequency Cepstral Coefficients) or Mel features. A deep learning algorithm (such as ECAPA-TDNN) can then be used to extract a voiceprint embedding vector from these MFCCs or Mel features. This voiceprint embedding vector is then compared with a registered user's voiceprint template, and the similarity between the two is calculated. This similarity is then used as the corresponding first voiceprint confidence level.

[0062] Similarly, when there are multiple registered users, the first voiceprint confidence between the detected person and each registered user can be obtained through the above calculation.

[0063] In step S140, a confidence reliability score is determined based on the environmental parameters.

[0064] Among them, environmental parameters can be various physical quantities of the environment outside the door that are collected, which may include light intensity, noise level, and the number of people outside the door, etc. These parameters may affect the accuracy and reliability of recognition.

[0065] The confidence reliability score can be a quantitative score that comprehensively evaluates the reliability of the recognition process based on environmental parameters, which can reflect the support degree of the current environment for the aforementioned confidence recognition.

[0066] In this embodiment, a comprehensive score can be made based on the collected environmental parameters to determine the confidence reliability score. In one example, an initial value can be set in advance, and then the initial value can be adjusted based on the environmental parameters to obtain the corresponding confidence reliability score. For example, when the light is too dim (such as Lux < 100), the initial value can be lowered according to a predetermined step size, otherwise, the initial value can be increased. When the background noise is too loud (such as > 70dB), the initial value can be lowered according to a predetermined step size, otherwise, the initial value can be increased. When there are multiple people outside the door (i.e., two or more), the initial value can be lowered according to a predetermined step size, otherwise, the initial value can be increased.

[0067] In this way, by comprehensively considering the impact of environmental factors on recognition, the system can automatically perceive and adapt to various complex environmental conditions in different environments, thereby improving the environmental adaptability and robustness of the intelligent door system.

[0068] Please continue to refer to Figure 1 In step S150, the total confidence score of the detected person and the registered user is determined based on the face confidence, behavior confidence, first voiceprint confidence and the confidence reliability score.

[0069] In this embodiment, after calculating the facial confidence, behavioral confidence, first voiceprint confidence, and confidence reliability score, a comprehensive calculation can be performed based on the above information to determine a final total confidence score. It should be understood that when there are multiple registered users, the above calculation can obtain a total confidence score (which can be a value between 0 and 1) for each registered user of the detected person. The larger the total confidence score corresponding to a registered user, the greater the possibility that the detected person is that registered user.

[0070] In this way, by comprehensively considering multiple confidence levels and environmental factors, the degree of match between the detected person and the registered user can be more comprehensively evaluated, effectively reducing the misidentification rate and improving the accuracy of identity recognition.

[0071] In step S160, the smart door is controlled to open or remain closed according to the total confidence score.

[0072] In this embodiment, the calculated total confidence score can be compared with a pre-set threshold. If the total confidence score is greater than or equal to the threshold, it indicates that the recognition is successful, and an open signal can be sent to the actuator of the smart door to perform the door opening action. Otherwise, no signal can be sent to the actuator, or a keep-closed signal can be sent to the actuator to keep the smart door closed.

[0073] In one example, after each recognition, whether the recognition is successful or failed, the corresponding door opening or door opening refusal event and related recognition information can be recorded for subsequent reference.

[0074] So, based on Figure 1 The illustrated embodiment combines multimodal information such as facial recognition, voiceprint detection, and behavioral analysis to achieve multi-factor cross-validation, effectively preventing counterfeit attacks such as 3D facial masks or fingerprint films. Compared with a single recognition method, the false recognition rate is significantly reduced, thereby improving the security and reliability of the smart door system.

[0075] In addition, the confidence reliability score is determined according to the environmental parameters and incorporated into the calculation of the total confidence score, so that it can adapt to different environmental conditions, improve the accuracy and stability of recognition, and ensure the normal operation of the smart door in various environments.

[0076] Regarding step S150, in some embodiments of the present disclosure, it may include: Figure 2 Steps S151 to S152 are shown.

[0077] In step S151, weight values ​​corresponding to the face confidence, the behavior confidence, the first voiceprint confidence, and the confidence reliability score are obtained.

[0078] In this embodiment, the weight values ​​α, β, γ, and δ can be initialized according to the system default settings or historical data, which correspond to the face confidence, behavior confidence, voiceprint confidence, and confidence reliability scores respectively.

[0079] In step S152, a weighted sum calculation is performed based on the face confidence, the behavior confidence, the first voiceprint confidence, the confidence reliability score and the weight value to obtain a total confidence score of the detected person corresponding to the registered user.

[0080] In this embodiment, the face confidence S_face, behavior confidence S_behavior, first voiceprint confidence S_voice, and confidence reliability score S_context obtained by the aforementioned calculations and the corresponding weight values ​​α, β, γ, and δ can be weighted and calculated according to the following formula to determine the total confidence score S_final of the detected person corresponding to the registered user: S_final = α·S_face + β·S_voice + γ·S_behavior + δ·S_context.

[0081] In this way, the confidence levels of multiple biometric and behavioral features are integrated and calculated, which increases the difficulty of forgery and deception and improves the security of the smart door system.

[0082] Based on the above embodiments, regarding step S160, in some embodiments of the present disclosure, step S160 may include the following: Figure 3 Steps S161 to S163 are shown.

[0083] In step S161, the maximum value of the total confidence scores of the detected person corresponding to each registered user is compared with the score threshold.

[0084] The score threshold may be a pre-set scoring system used to determine whether to allow the detected person to pass the identification. If the total confidence score is greater than or equal to the score threshold, the identification is considered successful; otherwise, the identification fails.

[0085] In this embodiment, the total confidence scores of the detected person and registered users can be compared, and a maximum value can be determined from the total confidence scores. The maximum value can then be compared with a preset score threshold.

[0086] In step S162, when the maximum value of the total confidence score is greater than or equal to the score threshold, the smart door is controlled to open.

[0087] In this embodiment, if the maximum value of the total confidence score is greater than or equal to the score threshold, the recognition is considered successful. At this time, an opening signal can be sent to the actuator of the smart door, thereby opening the smart door.

[0088] In step S163, when the maximum value of the total confidence score is less than the score threshold, the smart door is controlled to remain closed.

[0089] In this embodiment, when the minimum value of the total confidence score is less than the score threshold, it can be considered that the recognition has failed. A keep-closed signal can be sent to the actuator of the smart door, so that the smart door remains in a closed state.

[0090] In one example, when recognition fails, an alarm or prompt message may be triggered to notify the user or the security system.

[0091] In this way, by setting the score threshold, the door is allowed to open only when the total confidence score is high enough, effectively preventing illegal intrusion and improving the security of the smart door.

[0092] In some embodiments of the present disclosure, before performing weighted sum calculation based on the face confidence, the behavior confidence, the first voiceprint confidence, the confidence reliability score, and the weight value, the method further includes: The weight values ​​corresponding to the face confidence, the behavior confidence and / or the first voiceprint confidence are adjusted according to the environmental parameters, where the environmental parameters include at least one of light intensity, sound decibels and the number of people outside the door.

[0093] In this embodiment, based on the obtained initial weight values, the weight values ​​corresponding to the face confidence, behavior confidence, and / or first voiceprint confidence can be adjusted according to environmental parameters. For example, when the light intensity is lower than a preset threshold (e.g., 100 Lux), the face weight α is reduced because the accuracy of face recognition in low light decreases. When the ambient noise level exceeds a preset threshold (e.g., 70 dB), the voiceprint weight β is reduced because the accuracy of voiceprint recognition in a high-noise environment is affected. When multiple people are detected approaching at the same time, the voiceprint weight β and face weight α are reduced, and the behavior weight γ is increased because behavioral analysis can better distinguish individuals in multi-person scenarios.

[0094] Then, based on the adjusted weight values, subsequent weighted sum calculations can reduce misidentification caused by environmental interference.

[0095] In some embodiments of the present disclosure, in addition to adjusting weights based on environmental parameters, weights can also be adjusted based on user preferences. For example, if a user prefers facial recognition (which can be pre-set), the weight corresponding to facial recognition can be increased. If a user prefers voiceprint recognition, the weight corresponding to voiceprint detection can be increased, and so on.

[0096] In some embodiments of the present disclosure, the following may also be included: Figure 4 Steps S410 to S430 are shown.

[0097] In step S410, when it is detected that a person is approaching the door, audio information inside the door is collected.

[0098] In this embodiment, an infrared sensor or a distance sensor may also be installed on the inside of the smart door. When the sensor detects that someone is approaching the door, it may trigger the audio array to obtain audio information inside the door.

[0099] In step S420, semantic recognition and voiceprint detection are performed based on the audio information inside the door to determine the semantic information and the second voiceprint confidence between the detected person inside the door and the registered user.

[0100] Semantic recognition can be the process of understanding and parsing the voice content in audio information to identify the instructions or information in the voice.

[0101] In this implementation, voice activity detection technology is used to extract valid speech segments from the collected audio information within the door, removing silence and background noise. This processed audio information is then fed into a semantic recognition model, which, based on deep learning techniques (such as Transformer and BERT), analyzes the instructions or information (i.e., semantic information) contained in the speech.

[0102] At the same time, voiceprint features can be extracted from the audio information to determine the second voiceprint confidence between the detected person inside the door and the registered user. The specific processing method can be referred to as described above, and this disclosure will not go into details here.

[0103] In step S430, when the voice information is the target semantics and the second voiceprint confidence is greater than the voiceprint confidence threshold, the smart door is controlled to open.

[0104] In this embodiment, after obtaining the semantic information corresponding to the audio information inside the door and the second voiceprint confidence level, a judgment can be made based on these two. If the semantic information is the target semantics (e.g., "open the door," "I want to go out," etc.), and the second voiceprint confidence level is greater than or equal to a preset voiceprint confidence level threshold, it can be determined that the registered user wants to leave. Therefore, an opening signal can be sent to the actuator of the smart door to execute the door opening action.

[0105] In this way, through the dual verification of semantic recognition and voiceprint detection, it can correctly identify whether the user has left home, while improving the accuracy of identity recognition and system security, and reducing the possibility of misidentification. Moreover, the user does not need to manually open the door, which improves the convenience of the smart door.

[0106] In some embodiments of the present disclosure, the following may also be included: Figure 5 Steps S510 to S520 are described in detail as follows.

[0107] In step S510, during the closing process of the smart door, it is detected whether there is an object in the door gap area.

[0108] Among them, the door gap area can be the gap area between the door body and the door frame during the closing process of the smart door.

[0109] In this embodiment, when the smart door is closing, the infrared sensor array can be activated to detect the door gap area to determine whether there is an object in the door gap area (such as a user's hand, foot, or other object). In one example, a fixed frequency (such as 10 detections per second) can be set for real-time detection.

[0110] In step S520, when there is an object in the door gap area or the motor driving current of the smart door is greater than or equal to the current threshold, the smart door is controlled to stop closing and open in the reverse direction.

[0111] Among them, the motor driving current can be the current required by the motor of the smart door during operation, which can reflect the load condition of the motor.

[0112] In this embodiment, the smart door's motor drive current can be synchronously monitored, and the current value of each detection can be recorded. If the infrared sensor detects the presence of an object in the door gap, it is considered that the gap is obstructed or the user may be at risk of being pinched. If the motor drive current is greater than or equal to a preset current threshold, it indicates that the motor load is excessive and the door gap may be obstructed.

[0113] When an obstruction is detected in the door gap, a stop signal can be sent to the smart door's drive motor to stop the door from closing. The motor can then be driven in reverse, causing the smart door to open in the opposite direction for a certain distance or duration (e.g., 1 second) to avoid pinching injuries.

[0114] When it is determined that there is no obstruction in the door gap area, the closing action can continue until the smart door is completely closed.

[0115] In one embodiment, the system may record an anti-pinch event log for subsequent review.

[0116] This dual detection mechanism, through real-time monitoring of the door gap area and motor drive current, improves the accuracy of anti-pinch detection, ensuring reliable detection of obstructions in all situations, effectively preventing injuries to hands, feet, or other objects, and enhancing the safety of smart doors. When an obstruction is detected, the door immediately stops closing and reverses to open, reducing the risk of injury to the user or damage to property.

[0117] In some embodiments of the present disclosure, it also includes Figure 6 Steps S610 to S620 are shown.

[0118] In step S610, human behavior detection is performed based on the image information to determine the behavior detection result of the detected person outside the door. The human behavior detection result is used to characterize whether the detected person outside the door has abnormal behavior, and the abnormal behavior includes at least one of wandering outside the door and tailgating.

[0119] In this implementation, a camera installed on a smart door can capture real-time image information of the area outside the door. This captured image information can be input into a behavior detection model, which can use deep learning technologies (such as TCN) to identify the behavioral characteristics and trajectory of people.

[0120] Next, based on the behavioral features and trajectories output by the behavior detection model, it can be used to determine whether there is abnormal behavior. For example, identification of loitering outside a door can be defined as a person remaining in a specific area outside the door (e.g., within 2 meters in front of the door) for a period exceeding a predetermined field of view, and with a reciprocating movement trajectory. Tailgating identification can be defined as detecting an unauthorized person attempting to follow an authorized person within a certain period of time.

[0121] According to the above judgment results, the corresponding behavior detection results are output to indicate whether the detected person outside the door has abnormal behavior.

[0122] In step S620, when the detected person outside the door has abnormal behavior, the opening signal of the smart door is suppressed.

[0123] In this implementation, if the behavior detection result indicates that the person outside the door is behaving abnormally, the smart door's opening signal can be suppressed. This means that even if other recognition conditions (such as voiceprint or face recognition) are met, the door opening action will not be executed. At this time, an audible and visual alarm can be triggered to alert the homeowner or security system to the abnormal situation.

[0124] In one example, corresponding abnormal behavior events, such as time, personnel characteristics, and other information, may also be recorded for subsequent review.

[0125] In this way, abnormal behavior outside the door can be effectively identified, potential intruders or suspicious persons can be discovered in time, illegal intrusions can be prevented, and the safety of the living environment can be improved.

[0126] In some embodiments of the present disclosure, when a user leaves the house, behavior detection can be performed to determine whether the user turns back within a certain period of time (e.g., 3 seconds). If the user turns back, the smart door's opening signal is suppressed to prevent accidental opening.

[0127] In some embodiments of the present disclosure, there is also provided Figure 7The smart door shown in FIG. The smart door package integrates a multimodal sensor group 710, a main control unit 720 and an actuator 730.

[0128] The multimodal sensor group 710 may be a module integrating multiple sensors, including a binocular camera, an audio array, and an infrared sensor, so as to collect images, audio, and environmental parameters.

[0129] In one embodiment, a sensor installation slot, a device cavity, and a line routing slot may be reserved inside the door frame of the smart door, thereby realizing concealed installation of the device.

[0130] The main control unit 720 is the core processing unit of the smart door. It can adopt an ARM architecture SoC (System on Chip) and integrate an AI computing unit, providing high-performance processing capabilities. The main control unit 720 is electrically connected to the multimodal sensor group 710, so that it can execute the smart door control method described in any of the above embodiments based on the detection information of the multimodal sensor group 710.

[0131] Actuator 730 is electrically connected to main control unit 720 and can drive the smart door to open or close according to instructions from main control unit 720. In one example, actuator 730 can be a motor or door closer. Alternatively, the actuator can be a smart door closer, an electrically controlled door closing device with buffering and collision avoidance functions. It uses a brushless motor combined with a planetary reducer and has a door opening torque of ≥50 N·m.

[0132] In one embodiment, the door leaf may have built-in actuators, sensor arrays, and interactive devices (such as display screens, etc.), and the interior of the door leaf may be designed with an aluminum alloy frame and composite material panels to reduce the weight of the door leaf and facilitate electric control.

[0133] In some embodiments of the present disclosure, Figure 8 As shown, the smart door may further include an interaction module 740, which may be electrically connected to the main control unit 720. The interaction module may include at least one of an inner door touch screen, an outer door display screen, and a status indication unit.

[0134] The inner door touchscreen can be a touchscreen on the inside of the door, used to display information (such as door opening events, abnormal behavior events, etc.) and provide a user interface. The outer door display can be installed at a suitable location on the outside of the door to display relevant information outside the door, such as recognition results and environmental parameters.

[0135] The status indicator unit may include an LED light bar and / or an audio prompter to visually display the status of the smart door (open, closed, and when a specific event occurs). For example, when an anti-pinch event occurs, the LED light bar may turn red and the audio prompter may emit a certain sound prompt to remind the user to pay attention to safety.

[0136] Through the inner door touch screen and outer door display, users can intuitively view information both inside and outside the door, making operation and understanding easier. The status indicator unit uses color and sound prompts to keep users informed of the status of the smart door, enhancing the user experience.

[0137] In some embodiments of the present disclosure, the interaction module 740 may support remote connection with mobile devices (such as smartphones, tablets, etc.) Users can view door information, receive notifications, and remotely control the smart door through the mobile device.

[0138] In some embodiments of the present disclosure, Figure 9 As shown, the smart door can also include a power management module 750, which includes a main power link, a backup power supply and a power switching circuit, wherein the main power link is connected to the mains power, and the power switching circuit is used to switch to the backup power supply for power supply when the mains power is interrupted.

[0139] In this embodiment, the power management module 750 is the power supply and management system of the smart door, ensuring that the smart door can operate stably under different power conditions. Specifically, the main power link is the main power supply line of the smart door, which is usually connected to the mains power, which can convert the 220V mains power into 24V to power the various components of the smart door. The backup power supply is an auxiliary power supply that provides power to the smart door when the mains power is interrupted, which can be a rechargeable lithium battery pack. The power switching circuit can be a circuit that automatically switches the main power link and the backup power supply when the mains power is normal or interrupted.

[0140] When the mains power is normal, the main power link provides power to all components of the smart door (such as the multimodal sensor group, main control unit, actuators, etc.). When the mains power is interrupted, the power switching circuit detects when the main power voltage drops below a set threshold and automatically switches to the backup power source, which continues to power the smart door.

[0141] In some embodiments of the present disclosure, when switching to a backup power source, only the main modules of the smart door (such as the multimodal sensor group 710, the main control unit 720, and the actuator 730) can be powered, while the interaction module 740 is powered off. This can save power while ensuring the basic functions of the smart door are realized.

[0142] The present disclosure also provides an electronic device. Figure 10 A schematic diagram showing a hardware implementation using a processing system is shown.

[0143] like Figure 10 As shown, the hardware structure of electronic device 1000 can be implemented using a bus architecture. The bus architecture can include any number of interconnecting buses and bridges, depending on the specific application and overall design constraints of the hardware. Bus 1100 connects various circuits including one or more processors 1200, memory 1300, and / or hardware modules. Bus 1100 can also connect various other circuits 1400 such as peripheral devices, voltage regulators, power management circuits, external antennas, etc. Bus 1100 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Component Architecture (EISA) bus. Buses can be divided into address buses, data buses, control buses, etc. For ease of illustration, the figure only uses a single connecting line, but this does not mean that there is only one bus or only one type of bus.

[0144] The present disclosure also provides a readable storage medium, in which a computer program is stored, and the computer program is used to implement the above-mentioned method when executed by a processor. "Readable storage medium" can be any device that can contain storage, communication, dissemination or transmission programs for use in an instruction execution system, device or equipment or in combination with these instruction execution systems, devices or equipment. More specific examples of readable storage media include the following: an electrical connection portion with one or more wirings (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and editable read-only memory (EPROM or flash memory), an optical fiber device, and a portable read-only memory (CDROM), etc.

[0145] The present disclosure also provides a computer program product. The method of the present disclosure can be implemented in whole or in part using software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed, the process or function of the present disclosure is performed in whole or in part.

[0146] A computer program or instruction can be stored in a readable storage medium or transferred from one readable storage medium to another. For example, the computer program or instruction can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The readable storage medium can be any accessible medium or a data storage device such as a server or data center that integrates one or more accessible media. The accessible medium can be a magnetic medium such as a floppy disk, hard disk, or magnetic tape; an optical medium such as a digital video disk; or a semiconductor medium such as a solid-state drive. The computer-readable storage medium can be a volatile or non-volatile storage medium, or can include both volatile and non-volatile types of storage media.

[0147] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0148] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, apparatuses, electronic devices, and computer program products according to the present disclosure. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as the combination of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0149] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0150] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0151] In the description of this specification, the description with reference to the terms "one embodiment / method", "some embodiments / methods", "example", "specific example", or "some examples" means that the specific features, structures, or characteristics described in conjunction with the embodiment / method or example are included in at least one embodiment / method or example of the present disclosure. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment / method or example. Moreover, the specific features, structures, or characteristics described may be combined in a suitable manner in any one or more embodiments / methods or examples. In addition, those skilled in the art may combine and combine different embodiments / methods or examples described in this specification and the features of different embodiments / methods or examples, unless they are contradictory.

[0152] Those skilled in the art will appreciate that the above embodiments are merely intended to clearly illustrate the present disclosure and are not intended to limit the scope of the present disclosure. Other changes or modifications may be made based on the above disclosure, and such changes or modifications are still within the scope of the present disclosure.

Claims

1. An intelligent door control method based on multimodal information, characterized in that: include: When a person approaches the door, the system collects image information, audio information and environmental parameters outside the door; Performing face recognition and personnel behavior analysis based on the image information to determine the face confidence level and behavior confidence level between the detected person outside the door and the registered user; Performing voiceprint detection based on the audio information to determine a first voiceprint confidence level between the detected person and the registered user; Determining a confidence reliability score based on the environmental parameters; Determining a total confidence score of the detected person and the registered user based on the face confidence, behavior confidence, first voiceprint confidence, and the confidence reliability score; and The smart door is controlled to open or remain closed according to the total confidence score.

2. The method according to claim 1, wherein Determining the total confidence score of the detected person and the registered user based on the face confidence, behavior confidence, first voiceprint confidence, and the confidence reliability score, including: Obtaining weight values ​​corresponding to the face confidence, the behavior confidence, the first voiceprint confidence, and the confidence reliability score; A weighted sum calculation is performed based on the face confidence, the behavior confidence, the first voiceprint confidence, the confidence reliability score, and the weight value to obtain a total confidence score of the detected person corresponding to the registered user.

3. The method according to claim 2, wherein Controlling the smart door to open or remain closed according to the total confidence score includes: Comparing the maximum value of the total confidence scores of the detected person corresponding to each registered user with the score threshold; When the maximum value of the total confidence score is greater than or equal to the score threshold, controlling the smart door to open; When the maximum value of the total confidence score is less than the score threshold, the smart door is controlled to remain closed.

4. The method according to claim 2, wherein Before performing weighted sum calculation based on the face confidence, the behavior confidence, the first voiceprint confidence, the confidence reliability score, and the weight value, the method further includes: The weight values ​​corresponding to the face confidence, the behavior confidence and / or the first voiceprint confidence are adjusted according to the environmental parameters, where the environmental parameters include at least one of light intensity, sound decibels and the number of people outside the door.

5. The method according to claim 1, wherein Also includes: When a person approaches the door, the audio information inside the door is collected; Perform semantic recognition and voiceprint detection based on the audio information inside the door, and determine the confidence level of the semantic information and the second voiceprint between the detected person inside the door and the registered user; When the voice information is the target semantics and the second voiceprint confidence is greater than a voiceprint confidence threshold, the smart door is controlled to open.

6. The method according to claim 1, wherein Also includes: During the closing process of the smart door, detecting whether there is an object in the door gap area; When there is an object in the door gap area or the motor driving current of the smart door is greater than or equal to the current threshold, the smart door is controlled to stop closing and open in the reverse direction.

7. The method according to claim 1, wherein Also includes: Performing human behavior detection based on the image information to determine a behavior detection result of the detected person outside the door, wherein the human behavior detection result is used to indicate whether the detected person outside the door has abnormal behavior, wherein the abnormal behavior includes at least one of wandering outside the door and tailgating; When the detected person outside the door behaves abnormally, the opening signal of the smart door is suppressed.

8. A smart door, characterized in that: The smart door is integrated with: A multimodal sensor suite, including binocular cameras, an audio array, and infrared sensors; a main control unit, the main control unit being electrically connected to the multimodal sensor group, the main control unit being configured to execute the method according to any one of claims 1 to 7 based on detection information of the multimodal sensor group; An actuator is electrically connected to the main control unit and is used to drive the smart door to open or close according to the control instructions of the main control unit.

9. The smart door according to claim 8, characterized in that: Also includes: A power management module includes a main power link, a backup power supply and a power switching circuit, wherein the main power link is connected to the mains power, and the power switching circuit is used to switch to the backup power supply for power supply when the mains power is interrupted.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images