Door closing state detection method, device and system, robot and storage medium

Abstract

The invention relates to a door closing state detection method, device and system, a robot and a storage medium. The method comprises the following steps: acquiring a multi-modal sensing data stream in real time in the process that the mechanical arm unit executes a door closing action; according to the multi-modal sensing data stream, the door closing state of the intelligent equipment is detected in each modal, and a multi-modal detection result of the intelligent equipment is obtained; and performing fusion decision according to the multi-modal detection result to obtain the door closing state of the intelligent equipment. According to the method, intelligent detection of the door closing state can be achieved, the problem of misjudgment caused by a traditional single sensing detection mode is effectively avoided, efficiency and reliability are both considered, the misjudgment rate is fundamentally reduced, a more reliable intelligent decision is provided for the robot to execute intelligent home unmanned tasks, and the intelligent degree is higher.

Description

Technical Field

[0001] This invention relates to the field of smart homes, and in particular to a method, device, system, robot, and storage medium for detecting door closure status. Background Technology

[0002] In smart home and industrial automation scenarios, robots are widely used to assist or completely replace humans in unmanned tasks such as washing and drying. A key step is for the robot to automatically close the doors of smart devices like washer-dryers. Once the door is detected as closed, the washing and drying processes can begin, at which point the robot has completed its task. Therefore, detecting door closure is a crucial aspect of achieving smart home automation.

[0003] Currently, door closure detection mainly employs a single-sensor approach, such as using a Hall effect sensor or proximity switch mounted on the door to detect the contact between the door and the machine body. This detection method suffers from poor environmental adaptability, is susceptible to interference from operating conditions, lacks reliability, and has a limited detection dimension, resulting in a high false alarm rate. Summary of the Invention

[0004] To address one of the aforementioned technical deficiencies, this application provides a method, apparatus, system, robot, and storage medium for detecting door closure status. The technical solution includes:

[0005] A door closure state detection method is applied to a smart device equipped with a robotic arm unit, the robotic arm unit being used to perform a door closing action of the smart device, the method comprising:

[0006] Real-time acquisition of multimodal sensor data streams during the door-closing action of the robotic arm unit;

[0007] Based on the multimodal sensing data stream, the door-closing state of the smart device is detected in each mode to obtain the multimodal detection result of the smart device;

[0008] Based on the multimodal detection results, a fusion decision is made to obtain the door closure state of the smart device.

[0009] Optionally, the multimodal sensing data stream includes audio signals and force signals during the closing action performed by the robotic arm unit; correspondingly, the multimodal detection results include audio detection results and force detection results.

[0010] Based on the multimodal detection results, a fusion decision is made to obtain the door closure state of the smart device, including:

[0011] When both the audio detection result and the force detection result indicate that a door closing event exists during the process of the robotic arm unit performing the door closing action, the audio key features representing the door closing event are extracted from the audio signal, and the first timestamp corresponding to the audio key features is recorded.

[0012] Extract the key force features representing the door closing event from the force signal, and record the second timestamp corresponding to the key force features;

[0013] Determine whether the first timestamp and the second timestamp satisfy the timing consistency requirement;

[0014] When the first timestamp and the second timestamp satisfy the timing consistency, the door of the smart device is determined to be in a closed state; when the first timestamp and the second timestamp do not satisfy the timing consistency, the door of the smart device is determined to be in an open state.

[0015] Optionally, determining whether the first timestamp and the second timestamp satisfy the time sequence consistency includes:

[0016] Determine whether the time delay between the first timestamp and the second timestamp is within a preset timing window; if so, determine that the first timestamp and the second timestamp satisfy timing consistency; otherwise, determine that the first timestamp and the second timestamp do not satisfy timing consistency.

[0017] Optionally, the process of making a fusion decision based on the multimodal detection results to obtain the door closure state of the smart device further includes:

[0018] When only one of the audio detection results and the force detection results indicates that a door closing event exists during the door closing action of the robotic arm unit, the audio detection result and the force detection result are arbitrated according to the preset modal priority.

[0019] Optionally, the multimodal sensing data stream also includes visual signals during the closing action performed by the robotic arm unit; correspondingly, the multimodal detection result also includes visual detection results.

[0020] The process of performing a fusion decision based on the multimodal detection results to obtain the door closure state of the smart device also includes:

[0021] After the fusion decision is completed based on the audio detection result and the force detection result, a first-level verification signal is issued, and the result obtained by the fusion decision based on the audio detection result and the force detection result is determined as the first-level verification result;

[0022] When the first-level verification signal is received, the door closure status of the smart device is determined by fusing the first-level verification result and the visual detection result.

[0023] Optionally, the visual signals during the closing action of the robotic arm unit also include panel indicator light images; correspondingly, the visual detection results also include panel status detection results obtained based on the panel indicator light images;

[0024] The process of performing a fusion decision based on the multimodal detection results to obtain the door closure state of the smart device also includes:

[0025] After the fusion decision is completed based on the first-level verification result and the visual detection result, a second-level verification signal is issued, and the result obtained by the fusion decision based on the first-level verification result and the visual detection result is determined as the second-level verification result.

[0026] When the secondary verification signal is received, the door closure status of the smart device is determined by fusing the secondary verification result and the panel status detection result.

[0027] Optionally, the multimodal sensing data stream includes force signals and visual signals during the process of the robotic arm unit performing the door closing action; correspondingly, the multimodal detection results include force detection results and visual detection results.

[0028] Based on the multimodal detection results, a fusion decision is made to obtain the door closure state of the smart device, including:

[0029] When both the force detection result and the visual detection result indicate that a door closing event exists during the closing action of the robotic arm unit, the contact force information of the end of the robotic arm unit when it contacts the door handle of the smart device is extracted from the force signal, and the relative position information between the end of the robotic arm unit and the door handle of the smart device is extracted from the visual signal.

[0030] When the contact force information satisfies the first condition and the relative position information satisfies the second condition, the door closure state of the smart device is determined to be closed; otherwise, the door closure state of the smart device is determined to be open.

[0031] Optionally, the first condition is that the contact force information is greater than a preset contact force threshold, and the second condition is that the relative position information is less than a preset position deviation threshold.

[0032] Optionally, the process of making a fusion decision based on the multimodal detection results to obtain the door closure state of the smart device further includes:

[0033] When only one of the force detection results and the visual detection results indicates that a door closing event exists during the door closing action of the robotic arm unit, the force detection result and the visual detection result are arbitrated according to the preset modal priority.

[0034] Optionally, the multimodal detection results include state detection results under multiple modes;

[0035] Based on the multimodal detection results, a fusion decision is made to obtain the door closure state of the smart device, including:

[0036] Assign confidence weights to the state detection results for each modality;

[0037] The door closure state of the smart device is determined based on the state detection results under all modalities and the corresponding confidence weights.

[0038] Optionally, the process of making a fusion decision based on the multimodal detection results to obtain the door closure state of the smart device further includes:

[0039] The confidence weights for each modality are dynamically adjusted.

[0040] Optionally, the multimodal sensing data stream includes audio signals, force signals, and visual signals during the closing action of the robotic arm unit; correspondingly, the state detection results under multiple modes include audio detection results, force detection results, and visual detection results.

[0041] The confidence weights for each modality are dynamically adjusted, including:

[0042] The environmental signal-to-noise ratio is obtained based on the audio signal.

[0043] When the environmental signal-to-noise ratio is lower than a preset signal-to-noise ratio threshold, the confidence weight corresponding to the audio detection result is reduced, while the confidence weight corresponding to the force detection result and the visual detection result is increased.

[0044] Optionally, the multimodal sensing data stream includes audio signals and force signals during the closing action of the robotic arm unit; correspondingly, the state detection results under multiple modes include audio detection results and force detection results.

[0045] The confidence weights for each modality are dynamically adjusted, including:

[0046] Friction anomaly detection is performed based on the force sensor signal;

[0047] When an abnormal friction event occurs during the closing action of the robotic arm unit, the confidence weight corresponding to the audio detection result is increased, and the confidence weight corresponding to the force detection result is decreased.

[0048] Optionally, friction anomaly detection is performed based on the force sensor signal, including:

[0049] The peak force information during the door-closing action of the robotic arm unit is determined based on the force sensor signal.

[0050] Based on the peak force information, anomaly judgment is made on the peak force. When anomaly of peak force exists, it is determined that there is an abnormal friction event during the door closing action of the robotic arm unit.

[0051] And / or, based on the force sensor signal, perform friction anomaly detection, including:

[0052] The force curve morphology information of the robotic arm unit during the door-closing action is determined based on the force sensor signal.

[0053] Based on the force curve morphology information, anomalies in the force curve morphology are determined. When anomalies in the force curve morphology are found, it is determined that there is an abnormal frictional force event during the closing action of the robotic arm unit.

[0054] Optionally, the multimodal sensing data stream includes audio signals during the closing action performed by the robotic arm unit; correspondingly, the multimodal detection result includes audio detection results.

[0055] Based on the multimodal sensing data stream, the door-closing state of the smart device is detected in each modality to obtain the multimodal detection results of the smart device, including:

[0056] Feature extraction is performed on the audio signal to obtain audio features;

[0057] The audio features are matched with a preset door closing event feature template library. When a match is successful, the audio features are input into a preset audio recognition network model to calculate the model confidence.

[0058] When the confidence level of the model exceeds a preset confidence threshold, the audio detection result of the smart device indicates that a door closing event exists.

[0059] Optionally, the multimodal sensing data stream includes force signals during the closing action performed by the robotic arm unit; correspondingly, the multimodal detection results include force detection results.

[0060] Based on the multimodal sensing data stream, the door-closing state of the smart device is detected in each modality to obtain the multimodal detection results of the washer-dryer, including:

[0061] The force sensory signal is subjected to feature extraction to obtain force sensory features;

[0062] The force sensory features are input into a preset force sensory recognition network model, and the force sensory detection results are output.

[0063] Optionally, the multimodal sensing data stream includes visual signals during the process of the robotic arm unit performing the door closing action; correspondingly, the multimodal detection result includes visual detection results;

[0064] Based on the multimodal sensing data stream, the door-closing state of the smart device is detected in each modality to obtain the multimodal detection results of the washer-dryer, including:

[0065] The visual signal is subjected to feature extraction to obtain visual features;

[0066] The visual features are input into a preset visual recognition network model, and the visual detection results are output.

[0067] Furthermore, the present invention also provides a door closure state detection device, applied in an intelligent device equipped with a robotic arm unit, the robotic arm unit being used to perform the door closing action of the intelligent device, the device comprising:

[0068] The data acquisition module is used to acquire multimodal sensor data streams in real time during the process of the robotic arm unit performing the door closing action;

[0069] The state perception module is used to detect the door closing state of the smart device in each mode according to the multimodal sensing data stream, and obtain the multimodal detection result of the smart device;

[0070] The state decision module is used to make a fusion decision based on the multimodal detection results to obtain the door closure state of the smart device.

[0071] Furthermore, the present invention also proposes a door closure state detection system, characterized in that it includes:

[0072] A memory, wherein a computer program is stored;

[0073] A processor configured to run the computer program to perform the aforementioned door closure state detection method.

[0074] Furthermore, the present invention also proposes a robot, characterized in that it comprises:

[0075] A robotic arm unit used to perform the door-closing action of smart devices;

[0076] The sensing unit is used to collect and acquire multimodal sensing data streams during the process of the robotic arm unit performing the door closing action;

[0077] The aforementioned door closure status detection system is communicatively connected to both the robotic arm unit and the sensing unit, and is used for:

[0078] Receive the multimodal sensing data stream in real time;

[0079] Based on the multimodal sensing data stream, the door-closing state of the smart device is detected in each mode to obtain the multimodal detection result of the smart device;

[0080] Based on the multimodal detection results, a fusion decision is made to obtain the door closure state of the smart device.

[0081] Furthermore, the present invention also proposes a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the aforementioned door closure state detection method during runtime.

[0082] The technical solution provided by this invention has the following advantages:

[0083] When the robotic arm unit performs the door-closing action on the washer-dryer, it acquires multi-modal sensor data in real time—a multi-modal sensor data stream—which allows for examination of the door's state from multiple angles, enabling multi-dimensional data cross-validation and defect complementarity. It also enables real-time dynamic detection, capturing transient anomalies, avoiding static misjudgments, adapting to dynamic operating conditions, and improving environmental robustness. By detecting the door-closing state in each modality through the multi-modal sensor data stream, it can examine the door's closure status from a single perspective while facilitating subsequent multi-modal fusion decision-making. This provides redundant backup for door state detection, preventing detection failures due to a single modality's malfunction and improving detection reliability. Finally, the fusion decision based on multi-modal detection results, rather than relying on a single modality, effectively avoids the misjudgment problems associated with traditional single-sensor detection modes, balancing efficiency and reliability. This fundamentally reduces the misjudgment rate, providing more reliable intelligent decision-making for the robot to perform unmanned smart home tasks, resulting in a higher level of intelligence. Attached Figure Description

[0084] To more clearly illustrate the technical solutions in this application, the drawings used in the application description will be briefly introduced below. Obviously, the drawings described below are only some of the applications of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0085] Figure 1 This is a model structural diagram of the washer-dryer in this invention;

[0086] Figure 2 This is a flowchart of the door closure detection method in this invention;

[0087] Figure 3 This is a flowchart of the first implementation of the present invention for making fusion decisions based on multimodal detection results;

[0088] Figure 4 This is a flowchart of the second implementation method of the present invention for making fusion decisions based on multimodal detection results;

[0089] Figure 5 This is a flowchart of the third implementation method of the present invention for making fusion decisions based on multimodal detection results;

[0090] Figure 6 This is a flowchart of the fourth embodiment of the present invention for making fusion decisions based on multimodal detection results;

[0091] Figure 7 This is a flowchart of the fifth embodiment of the present invention for making fusion decisions based on multimodal detection results;

[0092] Figure 8 This is a structural diagram of the door closure state detection device in this invention.

[0093] Explanation of reference numerals in the attached figures:

[0094] 1. Robotic arm unit; 2. Door body. Detailed Implementation

[0095] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described application is only a part of, and not all, of the applications of the present invention. The present invention will be described in detail below with reference to the accompanying drawings and the applications. It should be noted that, unless otherwise specified, the applications and features described in the present invention can be combined with each other.

[0096] It should be noted that the terms "first," "second," etc., in the specification, claims, and drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0097] In this embodiment of the invention, the term "multiple" refers to two or more, and other quantifiers are similar.

[0098] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the various embodiments of the present invention will be described in detail below with reference to the accompanying drawings. However, those skilled in the art will understand that many technical details are presented in the embodiments of the present invention to facilitate a better understanding of the invention. However, the technical solutions claimed in the present invention can be implemented even without these technical details and various variations and modifications based on the following embodiments. The division of the following embodiments is for ease of description and should not constitute any limitation on the specific implementation of the present invention. The various embodiments can be combined with and referenced by each other without contradiction.

[0099] In utilizing robots to operate intelligent devices and assist humans in completing unmanned tasks, taking the operation of a washer-dryer as an example, a robotic arm unit or a dedicated door-opening mechanism is typically used to trigger the unlocking and opening of the door. Then, the gripping or suction tools on the robotic arm unit pick up clothes from a designated area one by one and precisely place them into the washing unit of the washer-dryer. Finally, the robotic arm unit closes the washer-dryer door, and when the door is tightly closed, the washing and drying process begins. In this process, closing the washer-dryer door using the robotic arm unit is a crucial step; ensuring the door is tightly closed is a necessary condition for starting the washing and drying process. However, current traditional technologies mainly rely on single-sensor detection methods, which have poor environmental adaptability, are easily affected by operating conditions, and lack reliability. For example, when the impact force of the robotic arm unit closing the door is large, detection errors are likely to occur. Furthermore, if the door structure is fatigued or the material deforms, it may accidentally open after being closed, which a single sensor cannot detect. At the same time, the single-sensor detection method has a limited detection dimension and a high false alarm rate.

[0100] Example 1

[0101] To address the aforementioned issues, this application provides the following door closure status detection method. In this embodiment, a washer-dryer is used as an example; however, the door closure status can also be applied to smart devices such as dishwashers and ovens, and this implementation does not impose specific limitations. For ease of description, a washer-dryer will be used as an example below, and its model structure diagram is shown below. Figure 1 As shown, the device includes a robotic arm unit 1 and a door 2. The robotic arm unit 1 is used to perform the closing action of the washer-dryer. However, this does not mean that door closure detection is limited to the washer-dryer.

[0102] The implementation details of the door closure state detection method are explained below. The following content is only for ease of understanding and is not necessary for implementing this solution.

[0103] like Figure 2 As shown, the door closure state detection method in this embodiment includes the following steps:

[0104] S100: Real-time acquisition of multimodal sensor data streams during the door-closing action performed by the robotic arm unit;

[0105] S200: Based on the multimodal sensing data stream, the door closing state of the smart device is detected in each mode to obtain the multimodal detection results of the smart device;

[0106] S300: Based on the multimodal detection results, a fusion decision is made to obtain the door closure status of the smart device.

[0107] When the robotic arm unit performs the door-closing action on the washer-dryer, it acquires multi-modal sensor data in real time—a multi-modal sensor data stream—which allows for examination of the door's state from multiple angles, enabling multi-dimensional data cross-validation and defect complementarity. It also enables real-time dynamic detection, capturing transient anomalies, avoiding static misjudgments, adapting to dynamic operating conditions, and improving environmental robustness. By detecting the door-closing state in each modality through the multi-modal sensor data stream, it can examine the door's closure status from a single perspective while facilitating subsequent multi-modal fusion decision-making. This provides redundant backup for door state detection, preventing detection failures due to a single modality's malfunction and improving detection reliability. Finally, the fusion decision based on multi-modal detection results, rather than relying on a single modality, effectively avoids the misjudgment problems associated with traditional single-sensor detection modes, balancing efficiency and reliability. This fundamentally reduces the misjudgment rate, providing more reliable intelligent decision-making for the robot to perform unmanned smart home tasks, resulting in a higher level of intelligence.

[0108] In this embodiment, the robotic arm unit includes a robotic arm body, a controller, and an end effector. The robotic arm body is a multi-joint arm, and its joints are typically equipped with torque sensors. The end effector can be a gripper or push rod adapted to a door handle, or it can be a vacuum suction cup or a magnetic suction cup. The controller includes components such as a main control chip, a communication module, and a memory.

[0109] In the first embodiment of this example, in S100, the multimodal sensing data stream includes audio signals and force signals during the process of the robotic arm unit performing the door closing action; correspondingly, in S200, the multimodal detection results include audio detection results and force detection results.

[0110] Specifically, the aforementioned audio signals are acquired in real time using an audio sensor. These signals include characteristic sounds during the door closing process, such as the sound of the latch engaging, the sound of the door and sealing strip fitting together, and abnormal friction sounds. The audio sensor is preferably a directional microphone array, which can focus on acquiring sounds from the direction of the door lock, suppressing ambient noise from the sides or back (such as the sound of robot joint rotation). This sensor can be installed on the end effector of the robotic arm unit, or on the wrist or base of the robotic arm body, facing the washer-dryer door lock area. When the robot performs the closing action, the end effector directly contacts the door, allowing for the acquisition of clear audio signals with characteristic closing sounds. Alternatively, it can be installed near the door lock, such as on the inside of the door frame corresponding to the latch, or near the door hinge, to accurately acquire key characteristic sounds such as the "click" sound of the latch engaging the lock hole.

[0111] Specifically, the aforementioned force signals are acquired in real time using torque sensors, pressure sensors, and force sensors. These signals include contact force information, pressure distribution information, and abnormal resistance information during the door closing process. The sensors can be installed at the joints of the robotic arm unit to monitor load changes during the closing process; they can also be installed on the end effector to accurately acquire the contact force at the moment of closing; and they can be installed on the edge of the door to monitor whether the door is fully closed or whether there is abnormal resistance.

[0112] It should be understood that the above audio detection results refer to the door closure status determined based on the audio signal, while the force detection results refer to the door closure status determined based on the force signal.

[0113] In the first embodiment of this example, S200 includes:

[0114] S211: Extract features from the audio signal to obtain audio features;

[0115] S212: Match the audio features with a preset door closing event feature template library. When the match is successful, input the audio features into a preset audio recognition network model and calculate the model confidence.

[0116] S213: When the confidence level of the model exceeds the preset confidence threshold, the audio detection result of the smart device is that a door closing event exists.

[0117] In S211, the extracted audio features include, but are not limited to, short-time zero-crossing rate, short-time energy, and Mel-frequency cepstral coefficients (MFCC). These features are used to characterize the instantaneous, sudden, and high-frequency characteristics of the "click" sound. By extracting these audio features, the key auditory signal "click" in the door closing event can be converted into quantifiable and distinguishable digital features, providing a reliable audio dimension basis for subsequent judgment of whether the door has been successfully locked. At the same time, extracting these audio features can filter out various noises in the audio signal (such as washer-dryer standby noise, robot mechanical movement noise, and other equipment operation noise), which helps to improve detection robustness.

[0118] Prior to S211, the method also includes:

[0119] S210: Preprocess the audio signal.

[0120] Preprocessing can further filter out steady-state noise in the environment (such as the sound of a washing machine motor, ambient human voices, etc.), while also enhancing sound sources from the direction of the door lock and suppressing interference from other directions.

[0121] Specifically, preprocessing includes, but is not limited to, adaptive filtering and adaptive beamforming.

[0122] After the audio features are extracted, the real-time audio features are first quickly screened using template matching mode to initially screen out key features that match the door closing event, which can improve the real-time response speed. Then, the machine learning mode is triggered for recognition. The generalization ability of the audio recognition network model in machine learning technology can be used for accurate recognition, which can enhance the robustness of the entire detection system and adapt to different brands and door lock types in self-service laundry scenarios, with a higher fault tolerance rate.

[0123] Specifically, in step S212, the successful closing sound of the washer-dryer door lock, such as a "click," can be pre-recorded, and its features can be stored as a template in the door closing event feature template library. The audio features extracted in step S211 are then quickly matched with the door closing event feature template library. If the similarity exceeds a preset threshold (e.g., 90%), for example, a similarity of 95%, a "match successful" is triggered; if the similarity does not exceed the preset threshold, for example, a similarity of 85%, a "match failed" is triggered. The similarity calculation can use a conventional method and is not restricted here.

[0124] It should be understood that the successful closing sound of door locks of different brands or types of washer-dryers may be the same or different. Therefore, the successful closing sound of door locks stored in the door closing event feature template library can be recorded for a large number of different brands or types of washer-dryers.

[0125] Specifically, in S212, the audio recognition network model can use a lightweight deep learning model (such as a one-dimensional convolutional neural network or a temporal convolutional network) for end-to-end recognition. This model can be pre-trained in the cloud or locally using a large number of "click" sound and non-"click" sound (such as collision sound, friction sound) samples, which can more accurately identify the target sound in the audio signal that matches the door closing event in complex environments, and the detection accuracy in audio modalities is high.

[0126] Specifically, in S212~S213, a lightweight network model for audio recognition can be constructed using the sequence "input audio features → convolutional layer → pooling layer → fully connected layer → output layer (Sigmoid activation function)". The Sigmoid activation function is applied to the original output value of the output layer to calculate the model confidence score, and the specific formula is as follows:

[0127] ;

[0128] in, denoted as model confidence, and z as the original output value of the output layer.

[0129] When the calculated model confidence exceeds the preset confidence threshold (e.g., 0.8), for example, when the original output value is 2.1 and the calculated model confidence is 0.89, and 0.89 > 0.8, the model judges that the probability of the current audio signal having a door closing event is 89%, and thus the audio detection result can be considered as having a door closing event.

[0130] In the first embodiment of this example, S200 further includes:

[0131] S214: Extract features from the force sensory signal to obtain force sensory features;

[0132] S215: Input the force sensory features into a preset force sensory recognition network model and output the force sensory detection result.

[0133] Through feature extraction and model recognition of the force signals described above, physical force signals can be converted into quantifiable digital features based on neural network learning technology. This provides robust, quantifiable, and interference-resistant data support for the detection of the door closure status of washer-dryer, thereby improving the reliability of the entire door closure status detection.

[0134] Specifically, during the closing process, the force signal may exhibit a specific pattern. In the initial stage when the door begins to move, the force may briefly increase due to overcoming friction or spring resistance. Then, during the door acceleration phase, the force gradually decreases or remains constant. Subsequently, during the deceleration phase as the door approaches closing, the force may fluctuate due to magnetic attraction, buffer devices, or manual deceleration (e.g., decreasing first and then increasing). During this process, a sharp peak or fluctuation may occur at the moment the door closes. Therefore, throughout the entire closing process, the force signal is a continuously changing temporal signal. Thus, the force feature extraction in S214 can perform temporal feature extraction, extracting features based on the dynamic changes in force during the closing action (including sudden increases, stabilization, and decay of force). The extracted force features include peak force, trough force, and rate of change of force. Meanwhile, the force signal can be converted from the time domain to the frequency domain through Fourier transform to extract force features of different frequency bands, including the dominant frequency energy distribution (specifically, the dominant frequency of mechanical vibration when the latch is engaged, whose frequency is usually concentrated in 50~200Hz) and the low frequency energy distribution (specifically, the static pressure energy when the door is closed, whose frequency is usually <50Hz).

[0135] Specifically, in S215, the force recognition network model can also adopt a lightweight temporal network model, thus providing a reliable force dimension basis for subsequent judgment on whether the door is successfully locked. The specific implementation process of inputting force features into the force recognition network model and outputting force detection results can adopt the same implementation method as the aforementioned method of inputting audio features into the audio recognition network model and outputting audio detection results, or it can adopt other implementation methods, such as clustering algorithms; no restrictions are placed here.

[0136] The force detection results are similar to the aforementioned audio detection results, including the presence and absence of door closing events.

[0137] Based on the audio detection results and force detection results obtained from the aforementioned steps, such as Figure 3 As shown, in the first embodiment of this example, S300 includes:

[0138] S311: When both the audio detection result and the force detection result indicate that a door closing event exists during the process of the robotic arm unit performing the door closing action, extract the audio key features representing the door closing event from the audio signal and record the first timestamp corresponding to the audio key features;

[0139] S312: Extract the key force features representing the door closing event from the force signal, and record the second timestamp corresponding to the key force features;

[0140] S313: Determine whether the first timestamp and the second timestamp satisfy the timing consistency requirement;

[0141] S314: When the first timestamp and the second timestamp satisfy the timing consistency, the door closure state of the smart device is determined to be closed; when the first timestamp and the second timestamp do not satisfy the timing consistency, the door closure state of the smart device is determined to be open.

[0142] In a successful door closing event, audio features (such as the key audio feature "click") and force-feedback features (such as the key force-feedback feature indicating the "loss of force" point where the latch crosses the dead point in the force-feedback signal) are highly correlated in time. Therefore, when both the audio detection result and the force-feedback detection result indicate the existence of a door closing event, this embodiment records the timestamps representing the key audio features and the key force-feedback features (including the first timestamp and the second timestamp) respectively, and performs a time sequence consistency check on the two timestamps. This enables temporal coupling analysis of audio and force, checking whether the door closing event indicated in the audio mode occurs within the reasonable time delay corresponding to the door closing event indicated in the force-feedback mode, thereby greatly eliminating interference from irrelevant noise (such as a sound similar to a door closing coming from a distance). The above embodiment significantly improves the accuracy and robustness of door closing state detection through intelligent decision-making in the audio + force-feedback fusion mode.

[0143] Specifically, in S311, the "click" sound in the audio signal is extracted and the timestamp t1 (i.e., the first timestamp) is recorded; in S312, key points such as the marked peak force and force drop in the force signal are extracted and their timestamps t2 are recorded; in S313, a timing consistency check is performed based on t1 and t2. If the check is successful, the door closing event indicated by both the audio detection result and the force detection result is a valid door closing event, that is, the output door closing state is closed; otherwise, it is an invalid door closing event, and the output door closing state is open.

[0144] Furthermore, S313 includes:

[0145] Determine whether the time delay between the first timestamp and the second timestamp is within a preset timing window; if so, determine that the first timestamp and the second timestamp satisfy timing consistency; otherwise, determine that the first timestamp and the second timestamp do not satisfy timing consistency.

[0146] The time delay between the first timestamp t1 and the second timestamp t2 is If the If the timing window is preset (e.g., 0.5s), the timing consistency check is satisfied; otherwise, it is not satisfied.

[0147] like Figure 3 As shown, S300 also includes:

[0148] S315: When only one of the audio detection results and the force detection results indicates that a door closing event exists during the door closing action of the robotic arm unit, the audio detection results and the force detection results are arbitrated according to the preset modal priority.

[0149] When only one of the audio and force modes shows a door closing event (i.e., the washer-dryer door is closed), it indicates a conflict between the two modes. In this case, arbitration of the detection results between the two modes based on the preset mode priority can further avoid misjudgments or omissions caused by environmental interference, sensor failures, or other factors in single-mode detection. This can provide more comprehensive and accurate information for fusion decision-making and optimize the rationality of fusion decision-making.

[0150] Specifically, the preset modal priority can be pre-set according to the actual situation. For example, if the priority of the audio modality is higher than that of the force modality, then when only one of the detection results in the audio modality or the force modality indicates that a door closing event exists, the detection result in the audio modality (i.e., the audio detection result) can be directly used as the output result. Alternatively, the detection in the lower-priority force modality can be performed again. For example, if the robotic arm unit performs a small-stroke, low-force double-pushing action, the force signal can be collected again and a force detection result can be obtained. Based on the newly obtained force detection result and the original audio detection result, a fusion decision is made again. If the two conflict multiple times during arbitration, it can be determined as a fault and reported.

[0151] like Figure 3 As shown, S300 also includes:

[0152] S316: When both the audio detection result and the force detection result indicate that there is no closing event during the closing action of the robotic arm unit, the door closing state of the smart device is determined to be the open state.

[0153] When the detection results in both the audio and force modes indicate that there is no door closing event, it means that the door of the washer-dryer is detected as not being closed in both modes. Therefore, the door closure state is directly determined to be open, thus realizing closed-loop detection of the door closure state in both the audio and force modes.

[0154] In the second embodiment of this example, in S100, the multimodal sensing data stream also includes visual signals during the process of the robotic arm unit performing the door closing action; correspondingly, in S200, the multimodal detection result also includes visual detection results.

[0155] Specifically, the aforementioned visual signals are acquired in real time using visual sensors, such as small cameras. One or more of these visual sensors can be mounted on the end effector of the robotic arm unit. They can accurately identify features such as the shape and position of the door and door handle, providing accurate information for the robotic arm unit to plan safe approach trajectories, effectively preventing collisions, in addition to providing sensor data under visual modalities. The visual sensors can also be mounted on the base of the robotic arm unit, facing the washer-dryer door. This allows for continuous and stable monitoring of the gap between the door and the door frame during the closing process, unaffected by the robotic arm's movements. Therefore, they can completely detect the edges of the gap, perform accurate edge detection and fitting, and comprehensively determine whether the door is completely and smoothly closed by calculating the gap width, providing reliable geometric evidence of the closing state. The visual sensors can also be mounted on the end effector or joint of the robotic arm unit, facing the washer-dryer's control panel. This allows for easy coverage of the control panel, facilitating the identification of various visual feedback triggered by successful door closing, such as indicator lights illuminating or specific icon changes, providing redundant data support for subsequent door closure status detection.

[0156] It should be understood that the aforementioned visual detection result refers to the door closure state determined based on visual signals. This can be the door closure state detected based on a door image or door gap image, or it can be the door closure state detected based on a panel indicator light image. Furthermore, in the second embodiment of this example, before S300, audio signals and force signals need to be detected separately to obtain audio detection results and force detection results. The specific processes for detecting audio signals and force signals are implemented in the same way as in the first embodiment described above, and will not be repeated here.

[0157] In the second embodiment of this example, S200 further includes:

[0158] S221: Perform feature extraction on the visual signal to obtain visual features;

[0159] S222: Input the visual features into a preset visual recognition network model and output the visual detection result.

[0160] Visual features can include the color characteristics of different door handles and the color characteristics of different indicator lights on the control panel, which can be statistically analyzed using color histogram processing. Alternatively, they can include the shape and texture features of the door handles and the control panel, extracted using edge detection algorithms. Similar to the aforementioned audio recognition network model and force recognition network model, a pre-defined visual recognition network model can provide robust, quantifiable, and interference-resistant data support for the detection of washer-dryer door closure status under a visual modality, further improving the accuracy and reliability of subsequent fusion decisions.

[0161] In the second embodiment of this example, after detecting the audio signal and the force signal respectively to obtain the audio detection result and the force detection result, a fusion decision is first made based on the audio detection result and the force detection result. The specific process is the same as in the first embodiment, including steps S321~S326 (the same as steps S311~S316 in the first embodiment). Then, a fusion decision is made in combination with the visual detection result, that is, as follows: Figure 4 As shown ( Figure 4 (Steps S321~S326 omitted), S300 also includes:

[0162] S327: After completing the fusion decision based on the audio detection result and the force detection result, a first-level verification signal is issued, and the result obtained by the fusion decision based on the audio detection result and the force detection result is determined as the first-level verification result;

[0163] S328: When the first-level verification signal is received, the door closure state of the smart device is determined by fusing the first-level verification result and the visual detection result.

[0164] The above implementation method performs fusion decision-making in three modalities: audio, force, and vision. The audio + force fusion decision is used as the first-level verification, and then combined with vision for the second-level verification. Specifically, it can be called the audio + force → vision fusion mode. This implementation method can start the visual modality detection for fine verification after receiving the first-level verification signal, realizing a two-level redundant verification with audio and force as the main means and vision as the auxiliary means. It can detect the door closure status with the assistance of visual signals when the audio signal or force signal is missing or abnormal, further improving the reliability of intelligent detection of door closure status. It can adapt to smart devices of different brands, models, and ages, and has strong robustness.

[0165] Specifically, in S328, when a first-level verification signal is received, the fusion decision based on the first-level verification result and the visual inspection result (i.e., second-level redundancy verification) includes the following four cases:

[0166] First, if the first-level verification result indicates that the door is closed and the visual detection result also indicates that a door closing event has occurred, it means that there is no conflict between the first-level verification and the second-level redundant verification under the visual modality. This means that the washer-dryer door is closed and the door is directly determined to be closed.

[0167] Second, if the first-level verification result indicates that the door is closed, but the visual detection result does not indicate that a door closing event has occurred, it means that there is a conflict between the first-level verification and the second-level redundant verification under the visual modality, and arbitration is required.

[0168] Third, if the first-level verification result indicates that the door is in an open state while the visual detection result indicates that a door closing event has occurred, it means that there is a conflict between the first-level verification and the second-level redundant verification under the visual modality, and arbitration is required.

[0169] Fourth, if the first-level verification result indicates that the door is in an open state and the visual detection result also indicates that there is no door closing event, it means that there is no conflict between the first-level verification and the second-level redundant verification under the visual modality. This means that the washer-dryer door is not closed tightly, and the door is directly determined to be in an open state.

[0170] During the arbitration process, the priorities of the audio + force fusion modality and the visual modality can be pre-set (i.e., the priorities of primary verification and secondary redundancy verification under the visual modality can be pre-set). For example, if the priority of secondary redundancy verification under the visual modality is higher than that of primary verification, the door closure state corresponding to the visual detection result can be directly output; alternatively, a preset retry mode can be executed, specifically by using a robotic arm unit to re-execute a small-stroke, small-force secondary door-pushing action, collecting audio and force signals again and re-fusing them to obtain a new primary verification result. Based on the newly obtained primary verification result and the visual detection result, a fusion decision is made again. If there are multiple conflicts between primary verification and secondary redundancy verification under the visual modality during arbitration, it can be determined as a fault and reported.

[0171] This implementation also includes the case where no first-level verification signal is received. In this case, it means that the first-level verification corresponding to the audio + force fusion decision has not been triggered or the output is ambiguous (e.g., sensor failure). In this case, the gate closure state indicated by the result of the second-level redundant verification in the visual modality (i.e., the visual detection result) can be directly output as the final detection result.

[0172] Based on the second embodiment described above, this embodiment also provides a third embodiment. In S100, the visual signal during the closing action of the robotic arm unit further includes a panel indicator light image; correspondingly, in S200, the visual detection result further includes a panel status detection result obtained based on the panel indicator light image.

[0173] It should be understood that in the third embodiment, the visual signal includes, on the one hand, an image of the door gap between the smart device's door and the door handle (i.e., the first type of visual signal), and the visual detection result obtained based on the door gap image is determined as the door gap state detection result. Following steps S327-S328 of the second embodiment, the door gap state detection result is used as a secondary redundancy verification and fused with the primary verification under the audio + force fusion modality for a fusion decision. On the other hand, the visual signal also includes panel indicator light images (the second type of visual signal; therefore, the visual signal in the third embodiment specifically includes two types of visual signals).

[0174] In the third embodiment, based on steps S331-S338 (which are implemented in the same way as the aforementioned steps S321-S328), as follows: Figure 5 As shown ( Figure 5 (Steps S331~S336 omitted), S300 also includes:

[0175] S339: After completing the fusion decision based on the first-level verification result and the visual detection result, a second-level verification signal is issued, and the result obtained by the fusion decision based on the first-level verification result and the visual detection result is determined as the second-level verification result;

[0176] S3310: When the secondary verification signal is received, the door closure state of the smart device is determined by fusing the secondary verification result and the panel state detection result.

[0177] The third implementation method, after the second-level redundancy verification, fuses the second-level verification result (i.e., the result obtained by fusing the first verification result with the visual detection result corresponding to the first type of visual signal) with the visual detection result corresponding to the second type of visual signal again. This is specifically called the audio + force → vision → vision fusion mode, which provides a more reliable third-level redundancy verification mechanism. It uses the state of the washer-dryer equipment itself for feedback confirmation, which is more in line with the actual situation of the washer-dryer, fundamentally avoids misjudgment, and significantly reduces the misjudgment rate.

[0178] In S3310, the panel status detection result is obtained based on the panel indicator light image, that is, the door closing event is determined based on the panel indicator light image. This can also be achieved through steps such as image preprocessing, feature extraction, and model judgment.

[0179] Specifically, in S3310, when a secondary verification signal is received, the fusion decision based on the secondary verification result and the panel status detection result (i.e., tertiary redundancy verification) includes the following three cases:

[0180] First, if the secondary verification result indicates that the door is closed, and the panel status detection result also indicates that there is a door closing event (such as the washer-dryer indicator light turning on or a specific icon changing), it means that there is no conflict between the secondary redundancy verification and the tertiary redundancy verification under the visual modality. This means that the washer-dryer door is closed, and the door is directly determined to be closed.

[0181] Second, if the secondary verification result indicates that the door is closed, but the panel status detection result does not indicate that a door closing event has occurred (e.g., the washer / dryer indicator light is not lit or a specific icon has not changed), or if the secondary verification result indicates that the door is open, but the panel status detection result indicates that a door closing event has occurred (e.g., the washer / dryer indicator light is lit or a specific icon has changed), it indicates that there is a conflict between the secondary redundancy verification and the tertiary redundancy verification under the visual modality, and arbitration is required.

[0182] Third, if the secondary verification result indicates that the door is in the open state while the panel status detection result also indicates that there is no door closing event, it means that there is no conflict between the secondary redundancy verification and the tertiary redundancy verification under the visual modality. This means that the washer-dryer door is not closed tightly, and the door is directly determined to be in the open state.

[0183] Similarly, in the above arbitration process, the priorities of secondary and tertiary redundancy verification can be preset, and arbitration is based on the preset priorities. This process is similar to the second implementation method mentioned above, and will not be described again here.

[0184] In this embodiment, the case where no secondary verification signal is received is also included. This means that the secondary redundancy verification is not triggered or the output is ambiguous (e.g., sensor failure, camera obstruction, etc.) or there are multiple arbitration conflicts. In this case, the door closure state indicated by the result of the tertiary redundancy verification in the visual modality (i.e., the panel state detection result) can be directly output as the final detection result.

[0185] This embodiment provides a fourth implementation method. In S200, the multimodal detection result includes the state detection results under multiple modes.

[0186] like Figure 6 As shown, S300 includes:

[0187] S341: Assign confidence weights to the state detection results for each modality;

[0188] S342: Determine the door closure state of the smart device based on the state detection results under all modalities and the corresponding confidence weights.

[0189] Unlike the third implementation (a multimodal hierarchical fusion decision-making mode), this implementation is a multimodal simultaneous fusion decision-making mode. By simultaneously assigning corresponding confidence weights to the state detection results under multiple modalities, and then combining all state detection results and confidence weights, the gate closure state is determined. This also realizes the simultaneous fusion of multiple modal perception and decision-making, which can effectively avoid misjudgment and ensure the robustness of the detection system.

[0190] It should be understood that if the number of modalities is two or more, the state detection results under multiple modalities can include state detection results under the audio modality (i.e., audio detection results) and the force modality (i.e., force detection results), or state detection results under the audio modality (i.e., audio detection results) and the visual modality (i.e., visual detection results), or state detection results under the force modality (i.e., force detection results) and the visual modality (i.e., visual detection results), or state detection results under the audio modality (i.e., audio detection results), the force modality (i.e., force detection results), and the visual modality (i.e., visual detection results). The state detection results under each modality can be obtained using methods similar to those described in the aforementioned embodiments, and will not be elaborated upon here.

[0191] Specifically, in S341, the state detection results of each modality can be assigned a confidence level of 0 to 1. For example, when the confidence level of the output model based on the audio recognition network model is 85% (that is, the probability of the audio signal indicating a door closing event is 85%), the audio confidence level can be directly set to 0.85; as another example, when a 1kHz frequency with an energy of 0.08 is detected in the audio signal under the audio modality (where the minimum energy of the sound when the latch is engaged with the door latch is 0.05), the audio confidence level can be set to 1 (that is, it means that a door closing event has been detected at this time).

[0192] Specifically, in S342, the gate closure state of the smart device can be determined by weighted fusion. Let C be the state detection result in the i-th mode. i Its corresponding confidence weight is w i The formula for calculating the closed state of the door is:

[0193] ;

[0194] Where N is the total number of modes, and C is the output value corresponding to the gate closure state.

[0195] When calculating the output value corresponding to the door's closed state according to the above formula, a judgment rule can be preset for this output value. For example, if C ≥ m1, the door is determined to be in a closed state; if m2 ≤ C < m1, a retry or secondary verification is triggered; if C < m2, the door is determined to be in an open state (i.e., the door failed to close). Here, m1 and m2 are set or adjusted according to the specific situation.

[0196] like Figure 6 As shown, S300 also includes:

[0197] S343: Dynamically adjust the confidence weights for each modality.

[0198] Assigning variable confidence weights to different modes better matches the dynamic operating conditions of the washer-dryer, resulting in stronger dynamic adaptability, effectively avoiding interference caused by dynamic operating conditions, and higher reliability.

[0199] The multimodal sensing data stream in S100 includes audio signals, force signals, and visual signals during the closing action of the robotic arm unit; correspondingly, in S200, the state detection results under multiple modes include audio detection results, force detection results, and visual detection results.

[0200] Furthermore, in the first embodiment of S343, it specifically includes:

[0201] S343A1: Obtain the ambient signal-to-noise ratio based on the audio signal;

[0202] S343A2: When the environmental signal-to-noise ratio is lower than the preset signal-to-noise ratio threshold, the confidence weight corresponding to the audio detection result is reduced, while the confidence weight corresponding to the force detection result and the visual detection result is increased at the same time.

[0203] The above implementation method is referred to as the audio + force + vision fusion mode. When the environmental signal-to-noise ratio is too low, it indicates that the reliability of the detection results in the audio mode is reduced under the interference of environmental noise. At this time, the confidence weight of the audio mode is reduced while the confidence weight of the force and vision modes is increased. This can more accurately estimate the degree of interference of environmental noise on the door closing audio signal, providing a reliable basis for adjusting the confidence weight of the detection results according to the signal-to-noise ratio. It is suitable for both quiet home environments and noisy public places, has strong environmental adaptability, and can dynamically adjust the confidence weight according to the changes in environmental noise, with strong real-time performance.

[0204] Specifically, in S343A1, the environmental signal-to-noise ratio (SNR) can be calculated based on the ambient noise power and the signal power of the "click" door closing sound in the audio signal. For example, if the ambient noise power of the audio signal is statistically P1, and the signal power of the "click" door closing sound in the audio signal is P2, then the environmental SNR is calculated according to the following formula:

[0205] ;

[0206] SNR stands for Environmental Signal-to-Noise Ratio.

[0207] Specifically, in S343A2, when the environmental signal-to-noise ratio is lower than the preset signal-to-noise ratio threshold, the confidence weight of the audio detection result is reduced, and the confidence weight of the force detection result and the confidence weight of the visual detection result are increased at the same time. This can be achieved according to preset rules or actual conditions, such as by linear adjustment. The specific linear adjustment coefficient is not limited here.

[0208] The multimodal sensing data stream in S100 includes audio signals and force signals during the closing action of the robotic arm unit; correspondingly, in S200, the state detection results under multiple modes include audio detection results and force detection results.

[0209] Furthermore, in the second embodiment of S343, it specifically includes:

[0210] S343B1: Detect frictional anomalies based on the force sensor signal;

[0211] S343B2: When an abnormal friction event occurs during the closing action of the robotic arm unit, the confidence weight corresponding to the audio detection result is increased, and the confidence weight corresponding to the force detection result is decreased.

[0212] The above implementation method represents another audio + force sensing fusion mode. Normal friction ensures smooth opening and closing of the washer-dryer door, while abnormal friction may cause the door to jam or suddenly close during opening and closing. It also causes the door hinges, sealing strips, and other components to bear greater stress, resulting in a significant difference in the force required to close the door compared to normal friction conditions. Therefore, when abnormal friction events occur, the reliability of the detection results under the force sensing mode decreases. By reducing its corresponding confidence weight, it better reflects the actual operating conditions of the washer-dryer, thus achieving higher reliability.

[0213] Preferably, S343B1 includes:

[0214] The peak force information during the door-closing action of the robotic arm unit is determined based on the force sensor signal.

[0215] Based on the peak force information, anomaly judgment is made on the peak force. When anomaly of peak force exists, it is determined that there is an abnormal friction event during the closing action of the robotic arm unit.

[0216] The above steps are one method for detecting abnormal friction. Peak force represents the maximum force detected by the force sensor during the opening and closing of the washer-dryer door. It reflects the maximum resistance that the washer-dryer door needs to overcome at the moment of opening and closing. This resistance is closely related to factors such as the weight of the door, the lubrication of the door hinges, and the tightness of the sealing strip. When friction is abnormal, the peak force often deviates from the normal range. For example, poor lubrication of the door hinges can lead to an increase in peak force. Therefore, peak force can accurately determine abnormal friction conditions.

[0217] Specifically, when judging peak force anomalies based on peak force information, a large amount of peak force data (e.g., 100 times) of normal door closing operations can be collected, and the average peak force and the standard deviation of peak force can be obtained respectively. Then, the upper threshold and lower threshold of peak force for judging peak force anomalies can be set based on the average peak force and the standard deviation of peak force. Finally, the peak force information determined by actual determination (e.g., transient contact force information when the force sensor collects data at a frequency greater than 1KHz) is compared with the upper threshold and lower threshold of peak force, and the result of the peak force anomaly judgment is output.

[0218] Preferably, S343B1 includes:

[0219] The force curve morphology information of the robotic arm unit during the door-closing action is determined based on the force sensor signal.

[0220] Based on the force curve morphology information, anomalies in the force curve morphology are determined. When anomalies in the force curve morphology are found, it is determined that there is an abnormal frictional force event during the closing action of the robotic arm unit.

[0221] The above steps represent another method for detecting abnormal friction. The force curve morphology describes the shape of the force as a function of time or displacement throughout the entire opening and closing process of the washer-dryer door. It encompasses the characteristics of the entire process from the initial change of force to reaching its peak and then gradually decreasing. The force curve morphology can reflect the trend and pattern of force changes during the opening and closing of the washer-dryer door. Different door conditions (such as normal opening and closing, jamming, obstruction by foreign objects, etc.) will lead to differences in the force curve morphology. An abnormal force curve morphology may also indicate a problem with friction. For example, multiple fluctuations in the force curve may indicate that the door is jamming during opening and closing, which is a manifestation of abnormal friction. Therefore, the abnormality of friction can also be accurately determined based on the force curve morphology.

[0222] Specifically, when judging force curve shape anomalies based on force curve shape information, a large amount of data on the rising slope of the force curve from normal door closing operations (e.g., 100 times) can be collected, and the average and standard deviation of the rising slope can be statistically obtained. Then, based on the average and standard deviation of the rising slope, the upper and lower thresholds of the rising slope for judging force curve shape anomalies can be set. Finally, the force curve shape information (specifically the rising slope, which refers to the average slope of the force curve from the start of the force to the peak point) is compared with the upper and lower thresholds of the rising slope to output the result of the force curve shape slope anomaly judgment.

[0223] It should be understood that the friction anomaly detection described in S343B1 of this embodiment can be implemented by either of the two implementation methods described above, that is, it can be determined by using only peak force information or by using only force curve shape information. It can also be implemented by combining both implementation methods. When combining the two implementation methods, the existence of a friction anomaly event can be determined when either a peak force anomaly or a force curve shape anomaly exists, or when both a peak force anomaly and a force curve shape anomaly exist simultaneously.

[0224] This embodiment also provides a fifth implementation method, wherein the multimodal sensing data stream includes force signals and visual signals during the process of the robotic arm unit performing the door closing action; correspondingly, the multimodal detection results include force detection results and visual detection results.

[0225] This implementation is a fusion decision-making mode of force and vision modality (referred to as the force + vision fusion mode). The state detection results of each modality can be obtained by a method similar to that of the aforementioned implementation, which will not be described in detail here.

[0226] Correspondingly, such as Figure 7 As shown, S300 includes:

[0227] S351: When both the force detection result and the visual detection result indicate that a door closing event exists during the process of the robotic arm unit performing the door closing action, the contact force information of the end of the robotic arm unit when it contacts the door handle of the smart device is extracted from the force signal, and the relative position information between the end of the robotic arm unit and the door handle of the smart device is extracted from the visual signal.

[0228] S352: When the contact force information satisfies the first condition and the relative position information satisfies the second condition, the door closure state of the smart device is determined to be closed; otherwise, the door closure state of the smart device is determined to be open.

[0229] Force signals reflect the mechanical interaction between the robotic arm and the door handle, playing a crucial role in determining whether the door has received sufficient force to close. Visual signals, on the other hand, visually display the relative position between the robotic arm's end effector and the door handle, accurately determining whether the door has reached the closed position. Therefore, force and visual signals are complementary. Intelligent fusion decision-making using these two modalities can more comprehensively and accurately determine the door's closure status, reducing potential misjudgments from single signals and improving the reliability of door status detection.

[0230] It should be understood that contact force information can include peak value, average value, and rate of change of the contact force, all of which reflect the mechanical characteristics of the robotic arm unit when it comes into contact with the door handle. Relative position information can include relative distance and relative angle, where relative distance can intuitively reflect whether the door is close to the closed position, while relative angle can provide information about the direction and degree of door closure.

[0231] Specifically, in S352, the first condition is that the contact force information is greater than a preset contact force threshold, and the second condition is that the relative position information is less than a preset position deviation threshold.

[0232] When the robotic arm unit performs the closing action, the contact force reaches a specific peak value at the moment the door closes. At the same time, the gap between the end of the robotic arm and the door handle (i.e., the door gap) also reaches a specific value. Therefore, based on the preset contact force threshold and the preset position deviation threshold, the closing event can be determined, and the door closure status can be accurately detected, effectively avoiding misjudgment that may be caused by a single signal.

[0233] In the fifth embodiment, such as Figure 7 As shown, S300 also includes:

[0234] S353: When only one of the force detection results and the visual detection results indicates that a door closing event exists during the process of the robotic arm unit performing the door closing action, the force detection result and the visual detection result are arbitrated according to the preset modal priority.

[0235] When only one of the force detection results and the visual detection result indicates the presence of a door closing event, it indicates a conflict between the two modal detection results. In this case, arbitration of the detection results under the two modalities is carried out based on the preset modal priority. This can further avoid misjudgment or omission caused by environmental interference, sensor failure, or other factors in single-modal detection, and can provide more comprehensive and accurate information for fusion decision-making, thereby optimizing the rationality of fusion decision-making.

[0236] The aforementioned arbitration can be carried out using methods similar to those described above; specific details will not be elaborated here.

[0237] like Figure 7 As shown, S300 also includes:

[0238] S354: When both the force detection result and the visual detection result indicate that there is no closing event during the closing action of the robotic arm unit, the door closing state of the smart device is determined to be the open state.

[0239] When the detection results in both the force and vision modes indicate that there is no door closing event, it means that the door of the washer-dryer is detected as not being closed in both modes. Therefore, the door closure state is directly determined to be the open state, realizing the closed-loop detection of the door closure state in the force + vision fusion mode.

[0240] Example 2

[0241] This application also provides a door closure state detection device, applied in an intelligent device equipped with a robotic arm unit, wherein the robotic arm unit is used to perform the door closing action of the intelligent device, such as... Figure 8 As shown, the device includes:

[0242] The data acquisition module is used to acquire multimodal sensor data streams in real time during the process of the robotic arm unit performing the door closing action;

[0243] The state perception module is used to detect the door closing state of the smart device in each mode according to the multimodal sensing data stream, and obtain the multimodal detection result of the smart device;

[0244] The state decision module is used to make a fusion decision based on the multimodal detection results to obtain the door closure state of the smart device.

[0245] In this embodiment, the door closure state detection device acquires multi-modal sensor data in real time through a data acquisition module when the robotic arm unit performs a door-closing action on a smart device (such as a washer-dryer). This multi-modal sensor data stream allows for multi-angle examination of the door's state, enabling cross-validation and defect complementarity of multi-dimensional data. It also enables real-time dynamic detection, capturing transient anomalies, avoiding static misjudgments, adapting to dynamic conditions, and improving environmental robustness. The state perception module detects the door closure state in each modality based on the multi-modal sensor data stream. This allows for single-angle examination of the door closure state and facilitates subsequent multi-modal fusion decision-making, achieving redundant backup of door state detection and preventing detection failure due to a single modality failure, thus improving detection reliability. Finally, the state decision module makes decisions based on the fusion of multi-modal detection results, rather than relying on a single modality. This effectively avoids the misjudgment problems caused by traditional single-sensor detection modes, balancing efficiency and reliability, fundamentally reducing the misjudgment rate, and providing more reliable intelligent decision-making for robots to perform unmanned smart home tasks, resulting in a higher level of intelligence.

[0246] The door closure state detection device described in this embodiment corresponds to the door closure state detection method described above. For details not covered in this embodiment, please refer to Embodiment 1 and... Figure 1 and Figure 7 The specific details will not be elaborated here.

[0247] Example 3

[0248] This application also provides a door closure status detection system, the system comprising:

[0249] The memory stores a computer program; the processor is configured to run the computer program to execute the aforementioned gate closure state detection method.

[0250] The door closure detection system provided in this application, through a computer program stored in memory and running on a processor, can make decisions based on the fusion of multimodal detection results, rather than relying on a single modality detection. This enables intelligent detection of door closure status, effectively avoiding the misjudgment problems caused by traditional single-sensor detection modes, balancing efficiency and reliability, fundamentally reducing the misjudgment rate, and providing more reliable intelligent decision-making for robots to perform unmanned smart home tasks, resulting in a higher level of intelligence.

[0251] This application also provides a robot, including:

[0252] A robotic arm unit used to perform the door-closing action of smart devices;

[0253] The sensing unit is used to collect and acquire multimodal sensing data streams during the process of the robotic arm unit performing the door closing action;

[0254] The aforementioned door closure status detection system is communicatively connected to both the robotic arm unit and the sensing unit, and is used for:

[0255] Receive the multimodal sensing data stream in real time;

[0256] Based on the multimodal sensing data stream, the door-closing state of the smart device is detected in each mode to obtain the multimodal detection result of the smart device;

[0257] Based on the multimodal detection results, a fusion decision is made to obtain the door closure state of the smart device.

[0258] When the robot uses its robotic arm to close the door of a smart device (such as a washer-dryer), it collects multimodal sensor data streams in real time through its sensing unit and transmits this data to the door closure status detection system. By utilizing the perception and fusion decision-making of the door closure status detection system, intelligent detection of the door closure status can be achieved. This effectively avoids the misjudgment problems caused by the traditional single-sensor detection mode, balancing efficiency and reliability, fundamentally reducing the misjudgment rate, and providing more reliable intelligent decision-making for the robot to perform unmanned smart home tasks, resulting in a higher level of intelligence.

[0259] This application also provides a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the above-described door closure state detection method at runtime.

[0260] The aforementioned storage medium stores a computer program capable of running a door closure detection method. Based on the operation of the computer program, intelligent detection of the door closure status can be achieved, effectively avoiding the misjudgment problem caused by the traditional single-sensor detection mode. It balances efficiency and reliability, fundamentally reducing the misjudgment rate, and providing more reliable intelligent decision-making for robots to perform unmanned smart home tasks, resulting in a higher level of intelligence.

[0261] In this embodiment, the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the computer device, connecting various parts of the entire computer device through various interfaces and lines.

[0262] The memory can be used to store computer programs and / or models. The processor enables the cleaning robot to perform various functions by running or executing the computer programs and / or models stored in the memory, and by calling the data stored in the memory.

[0263] The memory can primarily include a program storage area and a data storage area. The program storage area can store the operating system and at least one application program required for a function (such as sound playback, image playback, etc.); the data storage area can store data created based on user usage (such as audio data, video data, etc.). Furthermore, the memory can include high-speed random access memory, and may also include non-volatile memory, such as hard disks, RAM, plug-in hard disks, SmartMedia Cards (SMC), Secure Digital (SD) cards, Flash Cards, at least one disk storage device, flash memory device, or other volatile solid-state storage devices.

[0264] It should be understood that each block of a flowchart and / or block diagram, as well as combinations of blocks in a flowchart and / or block diagram, can be implemented by a computer program. These computer programs can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that instructions executable by the processor of the computer or other programmable data processing device generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0265] The aforementioned computer-readable storage medium may be a computer-readable memory that operates in a particular manner on a computer or other programmable data processing device, storing a computer program in such a computer-readable memory to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable device for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the functions specified in one or more boxes are used to achieve the cleaning of the cleaning area of ​​the cleaning base station.

[0266] The functions implemented by the processor during runtime, the functions implemented by the door closure state detection system in the robot, and the functions implemented by the computer program or instructions during runtime are all identical to the steps of the door closure state detection method described above. Therefore, for details not covered in this application, please refer to Embodiment 1 and... Figures 1 to 7 The specific details will not be elaborated here.

[0267] Obviously, the applications described above are only a part of this invention, not all of it. Based on the applications in this invention, those skilled in the art can make other variations or modifications without inventive effort, and all such variations or modifications should fall within the scope of protection of this invention.

Claims

1. A method for detecting the closed state of a door, characterized in that, The method, applied in a smart device equipped with a robotic arm unit for performing a door-closing action of the smart device, includes: Real-time acquisition of multimodal sensor data streams during the door-closing action of the robotic arm unit; Based on the multimodal sensing data stream, the door-closing state of the smart device is detected in each mode to obtain the multimodal detection result of the smart device; Based on the multimodal detection results, a fusion decision is made to obtain the door closure state of the smart device.

2. The method according to claim 1, characterized in that, The multimodal sensing data stream includes audio signals and force signals during the closing action performed by the robotic arm unit; correspondingly, the multimodal detection results include audio detection results and force detection results. Based on the multimodal detection results, a fusion decision is made to obtain the door closure state of the smart device, including: When both the audio detection result and the force detection result indicate that a door closing event exists during the process of the robotic arm unit performing the door closing action, the audio key features representing the door closing event are extracted from the audio signal, and the first timestamp corresponding to the audio key features is recorded. Extract the key force features representing the door closing event from the force signal, and record the second timestamp corresponding to the key force features; Determine whether the first timestamp and the second timestamp satisfy the timing consistency requirement; When the first timestamp and the second timestamp satisfy the timing consistency, the door of the smart device is determined to be in a closed state; when the first timestamp and the second timestamp do not satisfy the timing consistency, the door of the smart device is determined to be in an open state.

3. The method according to claim 2, characterized in that, Determining whether the first timestamp and the second timestamp satisfy the time sequence consistency includes: Determine whether the time delay between the first timestamp and the second timestamp is within a preset timing window; if so, determine that the first timestamp and the second timestamp satisfy timing consistency; otherwise, determine that the first timestamp and the second timestamp do not satisfy timing consistency.

4. The method according to claim 2, characterized in that, The process of performing a fusion decision based on the multimodal detection results to obtain the door closure state of the smart device also includes: When only one of the audio detection results and the force detection results indicates that a door closing event exists during the door closing action of the robotic arm unit, the audio detection result and the force detection result are arbitrated according to the preset modal priority.

5. The method according to claim 2, characterized in that, The multimodal sensing data stream also includes visual signals during the closing action performed by the robotic arm unit; correspondingly, the multimodal detection results also include visual detection results. The process of performing a fusion decision based on the multimodal detection results to obtain the door closure state of the smart device also includes: After the fusion decision is completed based on the audio detection result and the force detection result, a first-level verification signal is issued, and the result obtained by the fusion decision based on the audio detection result and the force detection result is determined as the first-level verification result; When the first-level verification signal is received, the door closure status of the smart device is determined by fusing the first-level verification result and the visual detection result.

6. The method according to claim 5, characterized in that, The visual signals during the closing action of the robotic arm unit also include panel indicator light images; correspondingly, the visual detection results also include panel status detection results obtained based on the panel indicator light images. The process of performing a fusion decision based on the multimodal detection results to obtain the door closure state of the smart device also includes: After the fusion decision is completed based on the first-level verification result and the visual detection result, a second-level verification signal is issued, and the result obtained by the fusion decision based on the first-level verification result and the visual detection result is determined as the second-level verification result. When the secondary verification signal is received, the door closure status of the smart device is determined by fusing the secondary verification result and the panel status detection result.

7. The method according to claim 1, characterized in that, The multimodal sensing data stream includes force signals and visual signals during the closing action of the robotic arm unit; correspondingly, the multimodal detection results include force detection results and visual detection results. Based on the multimodal detection results, a fusion decision is made to obtain the door closure state of the smart device, including: When both the force detection result and the visual detection result indicate that a door closing event exists during the closing action of the robotic arm unit, the contact force information of the end of the robotic arm unit when it contacts the door handle of the smart device is extracted from the force signal, and the relative position information between the end of the robotic arm unit and the door handle of the smart device is extracted from the visual signal. When the contact force information satisfies the first condition and the relative position information satisfies the second condition, the door closure state of the smart device is determined to be closed; otherwise, the door closure state of the smart device is determined to be open.

8. The method according to claim 7, characterized in that, The first condition is that the contact force information is greater than a preset contact force threshold, and the second condition is that the relative position information is less than a preset position deviation threshold.

9. The method according to claim 7, characterized in that, The process of performing a fusion decision based on the multimodal detection results to obtain the door closure state of the smart device also includes: When only one of the force detection results and the visual detection results indicates that a door closing event exists during the door closing action performed by the robotic arm unit, the force detection result and the visual detection result are arbitrated according to the preset modal priority.

10. The method according to claim 1, characterized in that, The multimodal detection results include state detection results under multiple modes; Based on the multimodal detection results, a fusion decision is made to obtain the door closure state of the smart device, including: Assign confidence weights to the state detection results for each modality; The door closure state of the smart device is determined based on the state detection results under all modalities and the corresponding confidence weights.

11. The method according to claim 10, characterized in that, The process of performing a fusion decision based on the multimodal detection results to obtain the door closure state of the smart device also includes: The confidence weights for each modality are dynamically adjusted.

12. The method according to claim 11, characterized in that, The multimodal sensing data stream includes audio signals, force signals, and visual signals during the closing action of the robotic arm unit; correspondingly, the state detection results under multiple modes include audio detection results, force detection results, and visual detection results. The confidence weights for each modality are dynamically adjusted, including: The environmental signal-to-noise ratio is obtained based on the audio signal. When the environmental signal-to-noise ratio is lower than a preset signal-to-noise ratio threshold, the confidence weight corresponding to the audio detection result is reduced, while the confidence weight corresponding to the force detection result and the visual detection result is increased.

13. The method according to claim 11, characterized in that, The multimodal sensing data stream includes audio signals and force signals during the closing action of the robotic arm unit; correspondingly, the state detection results under multiple modes include audio detection results and force detection results. The confidence weights for each modality are dynamically adjusted, including: Friction anomaly detection is performed based on the force sensor signal; When an abnormal friction event occurs during the closing action of the robotic arm unit, the confidence weight corresponding to the audio detection result is increased, and the confidence weight corresponding to the force detection result is decreased.

14. The method according to claim 13, characterized in that, Friction anomaly detection based on the force sensor signal includes: The peak force information during the door-closing action of the robotic arm unit is determined based on the force sensor signal. Based on the peak force information, anomaly judgment is made on the peak force. When anomaly of peak force exists, it is determined that there is an abnormal friction event during the door closing action of the robotic arm unit. And / or, based on the force sensor signal, perform friction anomaly detection, including: The force curve morphology information of the robotic arm unit during the door-closing action is determined based on the force sensor signal. Based on the force curve morphology information, anomalies in the force curve morphology are determined. When anomalies in the force curve morphology are found, it is determined that there is an abnormal frictional force event during the closing action of the robotic arm unit.

15. The method according to claim 1, characterized in that, The multimodal sensing data stream includes audio signals during the door-closing action performed by the robotic arm unit; correspondingly, the multimodal detection results include audio detection results. Based on the multimodal sensing data stream, the door-closing state of the smart device is detected in each modality to obtain the multimodal detection results of the smart device, including: Feature extraction is performed on the audio signal to obtain audio features; The audio features are matched with a preset door closing event feature template library. When a match is successful, the audio features are input into a preset audio recognition network model to calculate the model confidence. When the confidence level of the model exceeds a preset confidence threshold, the audio detection result of the smart device indicates that a door closing event exists.

16. The method according to claim 1, characterized in that, The multimodal sensing data stream includes force signals during the closing action performed by the robotic arm unit; correspondingly, the multimodal detection results include force detection results. Based on the multimodal sensing data stream, the door-closing state of the smart device is detected in each mode to obtain the multimodal detection results of the washer-dryer, including: The force sensory signal is subjected to feature extraction to obtain force sensory features; The force sensory features are input into a preset force sensory recognition network model, and the force sensory detection results are output.

17. The method according to claim 1, characterized in that, The multimodal sensing data stream includes visual signals during the process of the robotic arm unit performing the door closing action; correspondingly, the multimodal detection results include visual detection results; Based on the multimodal sensing data stream, the door-closing state of the smart device is detected in each mode to obtain the multimodal detection results of the washer-dryer, including: The visual signal is subjected to feature extraction to obtain visual features; The visual features are input into a preset visual recognition network model, and the visual detection results are output.

18. A door closure status detection device, characterized in that, An apparatus for use in smart devices equipped with a robotic arm unit, the robotic arm unit being used to perform a door-closing action of the smart device, the apparatus comprising: The data acquisition module is used to acquire multimodal sensor data streams in real time during the process of the robotic arm unit performing the door closing action; The state perception module is used to detect the door closing state of the smart device in each mode according to the multimodal sensing data stream, and obtain the multimodal detection result of the smart device; The state decision module is used to make a fusion decision based on the multimodal detection results to obtain the door closure state of the smart device.

19. A door closure status detection system, characterized in that, include: A memory, wherein a computer program is stored; A processor configured to run the computer program to perform the method as described in any one of claims 1 to 17.

20. A robot, characterized in that, include: A robotic arm unit used to perform the door-closing action of smart devices; The sensing unit is used to collect and acquire multimodal sensing data streams during the process of the robotic arm unit performing the door closing action; And the door closure state detection system as described in claim 19, which is communicatively connected to both the robotic arm unit and the sensing unit, for: Receive the multimodal sensing data stream in real time; Based on the multimodal sensing data stream, the door-closing state of the smart device is detected in each mode to obtain the multimodal detection result of the smart device; Based on the multimodal detection results, a fusion decision is made to obtain the door closure state of the smart device.

21. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, wherein the computer program is configured to execute the method described in any one of claims 1 to 17 when it is run.

Images