Control method and system for intelligently sensing multi-modal information
Through the intelligent sensing of multimodal information control method, combined with the status information and sound analysis of smart door locks and smart toilets, the toilet seat usage status is predicted, which solves the problem that users cannot ask for help in time under unexpected circumstances in the toilet, and improves the use of the toilet Security and data processing accuracy.
Patent Information
- Application Number
- CN202411051161.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-01
- Publication Date
- 2024-12-03
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
In existing toilets, if an unexpected situation occurs during use, it will be difficult for outsiders to provide assistance and it will be difficult for the use of the toilet seat in real time.
Using intelligent sensing multimodal information control method, through human-computer interaction terminals, communication with smart door locks and smart toilets, information such as door panel status, cushion pressure, flush events, etc. is collected and analyzed, and combined with sound information, the toilet seat usage status is predicted. And display inquiry information and abnormal prompts in abnormal situations to improve the safety and convenience of users and external personnel.
It improves the intelligence and accuracy of toilet usage data processing, ensures user safety, and improves the accuracy of user usage status judgment and the experience of inside and outside the bathroom.
Smart Images

Figure CN119068579A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the general data processing technology field of the Internet industry, and specifically to a control method and system for intelligently perceiving multimodal information. Background Art
[0002] In shopping malls, office buildings and other places, most toilets are equipped with mechanical door locks to determine whether there is someone in the toilet according to the color displayed on the color card on the outside of the door lock. However, it is difficult to know the specific usage of the toilet. At present, toilets are usually equipped with emergency call devices to facilitate toilet users to call for help when an accident occurs during the use of the toilet. However, if the toilet user is unable to call for help through the emergency call device due to an accident, outsiders will not be able to provide timely assistance. Summary of the invention
[0003] The embodiments of the present application provide a control method and system for intelligently perceiving multimodal information, in order to improve the accuracy and intelligence of toilet usage data processing, thereby ensuring the safety of users using toilets.
[0004] In a first aspect, an embodiment of the present application provides a control method for intelligently sensing multimodal information, which is applied to a human-computer interaction terminal in a state sensing system, wherein the state sensing system includes the human-computer interaction terminal arranged throughout the inner and outer sides of a door panel, an intelligent door lock arranged in a locking area of the door panel, and an intelligent toilet in a toilet seat corresponding to the door panel, wherein the human-computer interaction terminal is respectively connected to the intelligent door lock and the intelligent toilet for communication, and the human-computer interaction terminal includes an inner display screen, an outer display screen, and a sound receiving module and a broadcasting module, and the method includes:
[0005] receiving first status information of the door panel reported by the smart door lock, and receiving second status information reported by the smart toilet, wherein the first status information is used to characterize the open state and the closed state of the door panel, and the second status information is used to characterize the seat pressure information, the flushing event information, and the sanitary film rotation event information;
[0006] The sound receiving module collects sound information of the space in the inner direction of the door panel, wherein the sound information includes at least one of the following: user sound information and mobile phone sound information;
[0007] Determine a prediction result according to the first state information, the second state information and the sound information, wherein the prediction result is used to characterize the predicted usage state of the smart toilet, and the prediction result includes a normal usage state and an abnormal usage state;
[0008] When the prediction result is an abnormal usage state, displaying an inquiry message on the inner display screen, wherein the inquiry message is used to inquire the user whether an abnormality occurs;
[0009] In response to a first feedback operation for the inquiry information, first display information is displayed on the outer display screen, the first display information is used to display an abnormal state message, the abnormal state message is used to characterize an abnormal state, and the first feedback operation is a user response operation;
[0010] In response to a second feedback operation for the inquiry message, second display information is displayed on the outer display screen, the second display information is used to display an abnormal prompt message for other personnel to view, and the second feedback operation is a system response operation.
[0011] In a possible example, determining a prediction result according to the first state information, the second state information, and the sound information includes:
[0012] Constructing first feature data using a one-hot encoding mechanism for the first state information;
[0013] constructing second feature data using a multi-hot encoding mechanism for the second state information, and performing feature normalization processing on the second feature data to obtain updated second feature data;
[0014] Constructing third feature data according to the type of sound contained in the sound information;
[0015] constructing a target feature vector according to the first feature data, the updated second feature data, and the third feature data;
[0016] Calling a pre-trained usage status prediction model, taking the target feature vector as a model input, and obtaining a prediction probability that the usage status is normal;
[0017] The prediction result is determined according to the prediction probability and a preset probability threshold.
[0018] In a possible example, the usage status prediction model is a logistic regression binary classification model.
[0019] In a possible example, the functional formula of the usage status prediction model is as follows:
[0020] f(d)=1÷(1+e d ),
[0021] d=W1×X1+W2×X2+W3×X3+W0;
[0022] Among them, f(d) represents the predicted probability that the usage status is normal, X1 is the first feature data, X2 is the updated second feature data, X3 is the third feature data, W1 is the first preset coefficient corresponding to X1, W2 is the second preset coefficient corresponding to X2, W3 is the third preset coefficient corresponding to X3, and W0 is the bias coefficient.
[0023] In a possible example, the formula of the loss function LOSS of the usage state prediction model is as follows:
[0024]
[0025] Where N is the number of samples in the training set, i represents the i-th sample in N samples, y(i) represents the true probability in the sample, y′(i) represents the predicted probability of the model, λ represents the L2 regularization term coefficient, and Regularization(ω) represents the L2 regularization term.
[0026] In one possible example, the abnormality includes at least one of an equipment abnormality, a physical abnormality, and a judgment abnormality. The equipment abnormality refers to an abnormality in at least one of the smart door lock and the smart toilet. The judgment abnormality refers to an error in the conclusion that the predicted result is an abnormal usage state.
[0027] In a possible example, in response to the first feedback operation for the query information, displaying first display information on the outer display screen includes:
[0028] If the first feedback operation is to mark the device as abnormal, then the abnormal status message is a device abnormality prompt message;
[0029] If the first feedback operation is to mark a physical abnormality, the abnormal state message is the abnormal prompt message;
[0030] If the first feedback operation is marking the judgment as abnormal, the abnormal status message is a usage duration update message.
[0031] In a possible example, when the prediction result is a normal use state, the method further includes:
[0032] Displaying a usage time setting interface on the inner display screen;
[0033] In response to a first setting operation, determining a target usage duration, wherein the first setting operation is a user response operation;
[0034] In response to a second setting operation, the target usage duration is determined according to an average usage duration obtained by pre-statistically analyzing historical usage duration data, and the second setting operation is a system response operation.
[0035] In a possible example, after determining the target usage time, the method further includes:
[0036] Displaying usage time information on the inner display screen and the outer display screen until the target usage time timing ends, and controlling the smart door lock to perform a door opening operation when the target usage time timing ends, the usage time information including the used time information and the remaining usage time information;
[0037] Alternatively, the usage time information is displayed on the inner display screen and the outer display screen until it is detected that the user manually opens the smart door lock.
[0038] In a second aspect, an embodiment of the present application provides a state perception system, the state perception system comprising a human-machine interaction terminal arranged through the inner and outer sides of a door panel, a smart door lock arranged in a locking area of the door panel, and a smart toilet in a toilet seat corresponding to the door panel, the human-machine interaction terminal being respectively connected to the smart door lock and the smart toilet in communication, the human-machine interaction terminal comprising an inner display screen, an outer display screen, a sound receiving module and a sound broadcasting module;
[0039] The smart door lock is used to collect first status information of the corresponding door panel and report the first status information to the human-computer interaction terminal, wherein the first status information is used to represent the open state and the closed state of the door panel;
[0040] The smart toilet is used to collect the second state information and report the second state information to the human-computer interaction terminal, wherein the second state information is used to represent the seat pressure information, the flushing event information, and the sanitary film rotation event information;
[0041] The human-computer interaction terminal is used to receive the first status information of the door panel reported by the smart door lock, and to receive the second status information reported by the smart toilet;
[0042] The sound receiving module collects sound information of the space in the inner direction of the door panel, wherein the sound information includes at least one of the following: user sound information and mobile phone sound information;
[0043] Determine a prediction result according to the first state information, the second state information and the sound information, wherein the prediction result is used to characterize the predicted usage state of the smart toilet, and the prediction result includes a normal usage state and an abnormal usage state;
[0044] When the prediction result is an abnormal usage state, displaying an inquiry message on the inner display screen, wherein the inquiry message is used to inquire the user whether an abnormality occurs;
[0045] In response to a first feedback operation for the inquiry information, first display information is displayed on the outer display screen, the first display information is used to display an abnormal state message, the abnormal state message is used to characterize an abnormal state, and the first feedback operation is a user response operation;
[0046] In response to a second feedback operation for the inquiry message, second display information is displayed on the outer display screen, the second display information is used to display an abnormal prompt message for other personnel to view, and the second feedback operation is a system response operation.
[0047] It can be seen that in this embodiment, the control method of intelligently sensing multimodal information is applied to a human-machine interaction terminal in a state sensing system, wherein the state sensing system includes the human-machine interaction terminal arranged throughout the inner and outer sides of the door panel, the intelligent door lock arranged in the locking area of the door panel, and the intelligent toilet in the toilet seat corresponding to the door panel, wherein the human-machine interaction terminal is respectively connected to the intelligent door lock and the intelligent toilet in communication, and the human-machine interaction terminal includes an inner display screen, an outer display screen, a sound receiving module, and a broadcasting module. Specifically, the human-machine interaction terminal receives the first state information for the door panel reported by the intelligent door lock, and receives the second state information reported by the intelligent toilet, wherein the first state information is used to characterize the open state and the closed state of the door panel, and the second state information is used to characterize the cushion pressure information, the flushing event information, and the sanitary film rotation event information; and collects the sound information of the space in the inner direction of the door panel through the sound receiving module, and the sound information includes at least one of the following: user sound information, mobile phone sound information; the prediction result can be determined based on the first state information, the second state information, and the sound information, and the prediction result is used to characterize the predicted usage state of the intelligent toilet, and the prediction result is used to characterize the predicted usage state of the intelligent toilet. The prediction result includes a normal usage state and an abnormal usage state; when the prediction result is an abnormal usage state, an inquiry message is displayed on the inner display screen, and the inquiry message is used to inquire the user whether an abnormality occurs; and in response to a first feedback operation on the inquiry message, first display information is displayed on the outer display screen, and the first display information is used to display an abnormal status message, and the abnormal status message is used to characterize the abnormal status, and the first feedback operation is a user response operation; or, in response to a second feedback operation on the inquiry message, second display information is displayed on the outer display screen, and the second display information is used to display an abnormal prompt message for other personnel to view, and the second feedback operation is a system response operation.
[0048] It can be seen that the human-computer interaction terminal comprehensively predicts the use status of the smart toilet through the first state information, the second state information, and the sound information, and then interacts with the user by displaying the inquiry information on the inner display screen in an abnormal use state, and then updates the display content of the outer display screen to the first display information or the second display information according to the first feedback operation or the second feedback operation in response to the inquiry information. This is conducive to improving the intelligence and accuracy of the human-computer interaction terminal in processing the use data in the toilet, thereby improving the accuracy of judging the user's use status, and then improving the accuracy of the display content of the inner and outer display screens, and improving the use experience of users in the toilet and people waiting outside the toilet. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0050] Figure 1 It is a schematic diagram of the architecture of a state perception system provided in an embodiment of the present application;
[0051] Figure 2 It is a flowchart of a control method for intelligently sensing multimodal information provided by an embodiment of the present application;
[0052] Figure 3 This is a schematic diagram of an interface for displaying inquiry information for a single toilet seat in a state perception system provided in an embodiment of the present application;
[0053] Figure 4 This is a schematic diagram of an interface in which a single toilet seat displays first display information in a state perception system provided by an embodiment of the present application;
[0054] Figure 5 This is a schematic diagram of an interface in which a single toilet seat displays second display information in a state perception system provided by an embodiment of the present application;
[0055] Figure 6 This is a schematic diagram of an interface for displaying a usage time setting interface for a single toilet seat in a state perception system provided in an embodiment of the present application. DETAILED DESCRIPTION
[0056] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0057] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.
[0058] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.
[0059] The embodiments of the present application are described below with reference to the accompanying drawings.
[0060] The technical solution of this application can be applied to Figure 1 The state perception system 10 shown in the figure includes a human-machine interaction terminal 110 arranged through the inner and outer sides of the door panel, a smart door lock 120 arranged in the locking area of the door panel, and a smart toilet 130 in the toilet seat corresponding to the door panel. The human-machine interaction terminal 110 is respectively connected to the smart door lock 120 and the smart toilet 130 for communication. The human-machine interaction terminal 110 includes an inner display screen 111, an outer display screen 112, a sound receiving module 113, and a sound broadcasting module 114. Among them, the inner display screen 111 can support a touch function, so that the toilet seat can interact through the inner display screen 111.
[0061] Specifically, the inner display screen 111 and the outer display screen 112 are used to display information to the user in the toilet and the person outside the toilet respectively, the sound receiving module 113 is used to collect sound, and the broadcasting module 114 is used to play information. The human-computer interaction terminal 110 may also include a processing module (not shown), which is used to obtain and process various information data and perform corresponding actions. The following method can be specifically performed by the processing module of the human-computer interaction terminal 110.
[0062] Specifically, the smart door lock 120 supports electronic unlocking and locking, as well as manual unlocking and locking. The smart door lock 120 can collect first state information of the corresponding door panel and report the first state information to the human-computer interaction terminal 110, where the first state information is used to represent the open state and closed state of the door panel.
[0063] Specifically, the smart toilet 130 is provided with sensors, such as pressure sensors and infrared sensors, etc., for collecting the second state information of the needle and reporting the second state information to the human-computer interaction terminal 110. The second state information is used to characterize the seat pressure information, flushing event information, and sanitary film rotation event information.
[0064] Specifically, the human-computer interaction terminal 110 is used to receive the first status information of the door panel reported by the smart door lock 120, and to receive the second status information reported by the smart toilet 130; collect the sound information of the space in the inner direction of the door panel through the sound receiving module 113, and the sound information includes at least one of the following: user sound information, mobile phone sound information; determine the prediction result according to the first status information, the second status information and the sound information, and the prediction result is used to characterize the predicted usage status of the smart toilet 130, and the prediction result includes a normal usage status and an abnormal usage status; when the prediction result is an abnormal usage status, display an inquiry message on the inner display screen 111, and the inquiry message is used to inquire the user whether an abnormality occurs; in response to the first feedback operation on the inquiry information, display the first display information on the outer display screen 112, and the first display information is used to display an abnormal status message, and the abnormal status message is used to characterize the abnormal status, and the first feedback operation is a user response operation; in response to the second feedback operation on the inquiry message, display the second display information on the outer display screen 112, and the second display information is used to display an abnormal prompt message for other personnel to view, and the second feedback operation is a system response operation.
[0065] See also Figure 2 , Figure 2 is a flow chart of a control method for intelligently sensing multimodal information provided by an embodiment of the present application. The method can be applied in Figure 1 The human-computer interaction terminal in the state perception system shown in FIG. Figure 2 As shown, the control method of intelligently sensing multimodal information includes:
[0066] Step S210, receiving first status information of the door panel reported by the smart door lock, and receiving second status information reported by the smart toilet.
[0067] The first state information is collected by the smart door lock and reported to the human-computer interaction terminal, and the first state information is used to represent the open state and closed state of the door panel. The second state information is collected by the smart toilet and reported to the human-computer interaction terminal, and the second state information is used to represent the seat pressure information, flushing event information, sanitary film rotation event information, etc.
[0068] In the specific implementation, the smart door lock will report the current door lock switch state information (that is, the first state information) to the human-computer interaction terminal after the door lock state is switched. The smart toilet can collect and report all real-time information in real time or at preset intervals to ensure the integrity and reliability of the information. In this case, the information collected and reported may be the second state information, or the information may be empty. Alternatively, the smart toilet can collect data in real time and report the corresponding information (that is, the second state information) to the human-computer interaction terminal when a single event occurs, so as to reduce the redundancy of the data and improve the effectiveness of the reported data while ensuring the timeliness of data reporting.
[0069] Step S220: collecting sound information of the space on the inner side of the door panel through the sound receiving module.
[0070] The sound information includes at least one of the following: user sound information and mobile phone sound information. The space on the inner side of the door panel refers to the space on the side of the door panel facing the toilet installation in a single toilet stall.
[0071] In a specific implementation, the sound receiving module can collect the sound information of the space in the inner direction of the door panel in real time. Alternatively, the sound receiving module can start collecting the sound information of the space in the inner direction of the door panel after a certain preset event occurs. For example, when it is detected that the second state information includes the cushion pressure information, and it is determined that the user is sitting on the smart toilet cushion based on the cushion pressure information analysis, the sound receiving module can be called immediately or after a preset time to collect the sound information of the space in the inner direction of the door panel, and report it to the human-computer interaction terminal after the collection is completed.
[0072] Step S230: determining a prediction result according to the first state information, the second state information and the sound information.
[0073] The prediction result is used to characterize the predicted usage state of the smart toilet, and the prediction result includes a normal usage state and an abnormal usage state. Specifically, the normal usage state can be understood as the usage state corresponding to the user having no physical abnormalities in the toilet, and the abnormal usage state can be understood as the usage state corresponding to the user having physical abnormalities in the toilet.
[0074] In a specific implementation, the human-computer interaction terminal can set multiple time nodes for determining the prediction results, and determine the prediction results according to the first state information, the second state information, and the sound information collected corresponding to each time node, so as to improve the comprehensiveness of the user safety detection.
[0075] Exemplarily, the multiple time nodes may include a first node after a user enters the toilet after a first period of time, a second node after a flush event information is determined from the second state information after a second period of time, etc. For example, for the first node, if the first state information changes from an open door state to a closed door state, it can be determined that a user has entered the toilet, and after the first period of time, a prediction result can be determined based on the first state information, the second state information, and the sound information to determine whether the user has sat down to use the toilet normally. For another example, for the second node, after determining that the second state information contains flush event information after a second period of time, a prediction result can be determined based on the first state information, the second state information, and the sound information to determine whether the user has completed the use of the toilet normally.
[0076] Step S240, when the prediction result is an abnormal usage state, displaying an inquiry message on the inner display screen.
[0077] The inquiry information is used to inquire the user whether an abnormality occurs. The accuracy and reliability of abnormal result identification can be improved by further confirming the specific situation of the abnormal usage status by displaying the inquiry information in an interactive manner with the user.
[0078] For example, when the prediction result is confirmed as an abnormal usage state, the query information displayed on the inner screen may be Figure 3 The content of the first interface 30 is displayed on the inner display screen. Figure 3As shown, a plurality of compartments can be provided in the bathroom, and a single compartment is a toilet seat. A single toilet seat is provided with a smart toilet 130, a smart door lock 120 located on the door panel, an inner display screen 111, a sound receiving module 113, and a broadcasting module 114. Among them, when the user uses a single toilet seat, the inner display screen of the toilet seat can display inquiry information, that is, the first interface 30. The first interface 30 includes a first theme description control 310, a confirmation option control 320, and a countdown control 330. Among them, the confirmation option control 320 is an "A. Equipment abnormality" confirmation option control, a "B. Physical abnormality" confirmation option control, and a "C. No abnormality" confirmation option control. The countdown control 330 specifically includes a countdown prompt control 331 and a time display control 332.
[0079] Step S250: In response to a first feedback operation for the inquiry information, first display information is displayed on the outer display screen.
[0080] The first display information is used to display an abnormal state message, the abnormal state message is used to represent an abnormal state, and the first feedback operation is a user response operation. Specifically, the user response operation refers to an operation performed by a user.
[0081] In one possible example, the abnormality includes at least one of an equipment abnormality, a physical abnormality, and a judgment abnormality. The equipment abnormality refers to an abnormality in at least one of the smart door lock and the smart toilet. The judgment abnormality refers to an error in the conclusion that the predicted result is an abnormal usage state.
[0082] Specifically, the abnormality refers to the specific content corresponding to the abnormality in the inquiry information used to inquire about the user, that is, the inquiry information is used to inquire whether the user has any of the following situations: equipment abnormality, physical abnormality, and judgment abnormality. Equipment abnormality specifically refers to the situation where the smart door lock incorrectly recognizes the open and closed states of the door panel, and the smart toilet incorrectly collects seat pressure information, flushing event information, and sanitary film rotation event information. Physical abnormality refers to the situation where the user in the toilet seat is physically unwell. Judgment abnormality refers to the conclusion that the predicted result is an abnormal usage state, which is an error, that is, the human-computer interaction terminal determines the normal usage state as an abnormal usage state.
[0083] It can be seen that in this example, by specifically subdividing the abnormalities into several types of abnormalities, such as device abnormalities, physical abnormalities, and judgment abnormalities, the accuracy of abnormal situation identification can be improved, user needs can be more accurately identified, and the user experience can be improved.
[0084] In a specific implementation, within a preset time period after the inner display screen displays the inquiry information, if a user response operation to the inquiry information is detected, it is confirmed that there is a first feedback operation to the inquiry information, and the human-computer interaction terminal will display the first display information on the outer display screen in response to the first feedback operation. The operation performed by the user is used to confirm the corresponding abnormality type when an abnormality occurs, and the abnormality type includes device abnormality, physical abnormality, judgment abnormality, etc.
[0085] In one possible example, in response to a first feedback operation for the inquiry information, first display information is displayed on the outer display screen, including: if the first feedback operation is to mark a device abnormality, then the abnormal status message is a device abnormality prompt message; if the first feedback operation is to mark a body abnormality, then the abnormal status message is the abnormal prompt message; if the first feedback operation is to mark a judgment abnormality, then the abnormal status message is a usage time update message.
[0086] Specifically, if the operation performed by the user is for a device abnormality, the first feedback operation is determined to be marking the device abnormality, and the corresponding abnormal status message is determined to be a device abnormality prompt message. If the operation performed by the user is for a physical abnormality, the first feedback operation is determined to be marking the physical abnormality, and the corresponding abnormal status message is determined to be an abnormal prompt message. If the operation performed by the user is for judging an abnormality, the first feedback operation is determined to be marking the judgment abnormality. At this time, the usage time setting interface can be displayed on the inner display screen to obtain a usage time update message, and the corresponding abnormal status message is determined to be the usage time update message. The usage time corresponding to the usage time update message may be consistent with the initial time or inconsistent, and no further restrictions are made here.
[0087] Exemplarily, the first display information displayed on the outer display screen may be as follows: Figure 4 The content of the second interface shown, that is, the abnormal status message, may be as follows Figure 4 The content of the second interface is shown. Figure 4 As shown in FIG. 1 , a single toilet stall is provided with a smart toilet 130, a smart door lock 120 located on the door panel, and an outer display screen 112. Specifically, when the user is Figure 3 The confirmation option control 320 in the first interface 30 performs a confirmation operation (such as single click, double click, long press, etc.), that is, it is determined that there is a first feedback operation, and at this time, the first display information will be displayed on the outer display screen 112, that is, the second interface. Figure 3 If the confirmation operation is performed on the "A. Device abnormality" confirmation option control in the first interface 30, the Figure 4 The interface shown in (a) is determined as the second interface and is displayed on the outer display screen 112 corresponding to the toilet seat. Figure 4The interface shown in (a) includes a first prompt theme description control and a first prompt content control 420. The first prompt theme description control specifically includes a first prompt icon control 411 and a first prompt title control 412. For another example, if the user Figure 3 If the confirmation operation is performed on the "B. Physical Abnormality" confirmation option control in the first interface 30, the Figure 4 The interface shown in (b) is determined as the second interface and is displayed on the outer display screen 112 corresponding to the toilet seat. Figure 4 The interface shown in (b) includes a second prompt theme description control and a second prompt content control 440. The second prompt theme description control specifically includes a second prompt icon control 431 and a second prompt title control 432. For another example, if the user Figure 3 If the confirmation operation is performed on the "C. No abnormality" confirmation option control in the first interface 30, the Figure 4 The interface shown in (c) is determined as the second interface and is displayed on the outer display screen 113 corresponding to the toilet seat. Figure 4 The interface shown in (c) includes an estimated usage time display control 450 , a used time display control 460 , and an estimated waiting time display control 470 .
[0088] It can be seen that in this example, by displaying abnormal prompt messages with different contents on the outer display screen corresponding to different first feedback operations, the accuracy of the content displayed on the outer display screen can be improved, which is more conducive to other people outside the toilet to clearly understand the estimated waiting time and provide assistance to users in the toilet when necessary, thereby meeting the needs of different users.
[0089] Step S260: In response to a second feedback operation on the inquiry message, display second display information on the outer display screen.
[0090] The second display information is used to display an abnormal prompt message for other personnel to view, and the second feedback operation is a system response operation. Specifically, the system response operation refers to an operation performed by the system self-response.
[0091] In a specific implementation, within a preset time period after the query information is displayed on the inner display screen, if no user response operation to the query information is detected, the human-computer interaction terminal will confirm that there is a second feedback operation for the query information, and will display the second display information on the outer display screen in response to the second feedback operation. Specifically, the confirmation of the existence of the second feedback operation for the query information by the human-computer interaction terminal can also be understood as the confirmation by the human-computer interaction terminal that the current abnormal state is a physical abnormality. At this time, the second display information can be an abnormal prompt message, which notifies other people outside the toilet seat through the outer display screen that the user in the toilet seat has a physical abnormality, thereby improving the timeliness of the discovery of the user's physical abnormality.
[0092] In a specific implementation, an infrared sensor may also be provided in the toilet, and the infrared sensor is used to detect the user's body posture. Specifically, before the human-computer interaction terminal confirms the existence of a second feedback operation for the inquiry information, the infrared sensor may be enabled first to detect the user's body posture through the infrared sensor, so as to determine whether the user is fainting, asleep, etc. If so, it is determined that a second feedback operation exists. If it is determined that there is no fainting or other situation in combination with the infrared sensor, the preset time period is recalculated. If the user's response operation to the inquiry information is still not detected after the recalculated preset time period, it is determined that a second feedback operation exists. Among them, the length of the recalculated preset time period and the original preset time period may be the same or different, and no further restrictions are made here.
[0093] Exemplarily, the second display information displayed on the outer display screen may be as follows: Figure 5 The interface content is shown. Figure 5 As shown, a single toilet is provided with a smart toilet 130, a smart door lock 120 on the door panel, and an outer display screen 112. Specifically, if no user response operation to the query information is detected within a preset time period after the inner display screen displays the query information, it is confirmed that there is a second feedback operation. At this time, the human-computer interaction terminal will default to the selection of Figure 3 The "B. Physical Abnormality" confirmation option control in the first interface corresponds to the situation, so the third interface 50 will be displayed on the outer display screen 112 corresponding to the toilet seat, that is, Figure 5 The third interface 50 includes a second prompt theme description control 510 and a second prompt content control 520 . The second prompt theme description control 510 specifically includes a second prompt icon control 511 and a second prompt title control 512 .
[0094] It can be seen that in this embodiment, the control method of intelligently sensing multimodal information is applied to a human-machine interaction terminal in a state sensing system, wherein the state sensing system includes the human-machine interaction terminal arranged throughout the inner and outer sides of the door panel, the intelligent door lock arranged in the locking area of the door panel, and the intelligent toilet in the toilet seat corresponding to the door panel, wherein the human-machine interaction terminal is respectively connected to the intelligent door lock and the intelligent toilet in communication, and the human-machine interaction terminal includes an inner display screen, an outer display screen, a sound receiving module, and a broadcasting module. Specifically, the human-machine interaction terminal receives the first state information for the door panel reported by the intelligent door lock, and receives the second state information reported by the intelligent toilet, wherein the first state information is used to characterize the open state and the closed state of the door panel, and the second state information is used to characterize the cushion pressure information, the flushing event information, and the sanitary film rotation event information; and collects the sound information of the space in the inner direction of the door panel through the sound receiving module, and the sound information includes at least one of the following: user sound information, mobile phone sound information; the prediction result can be determined based on the first state information, the second state information, and the sound information, and the prediction result is used to characterize the predicted usage state of the intelligent toilet, and the prediction result is used to characterize the predicted usage state of the intelligent toilet. The prediction result includes a normal usage state and an abnormal usage state; when the prediction result is an abnormal usage state, an inquiry message is displayed on the inner display screen, and the inquiry message is used to inquire the user whether an abnormality occurs; and in response to a first feedback operation on the inquiry message, first display information is displayed on the outer display screen, and the first display information is used to display an abnormal status message, and the abnormal status message is used to characterize the abnormal status, and the first feedback operation is a user response operation; or, in response to a second feedback operation on the inquiry message, second display information is displayed on the outer display screen, and the second display information is used to display an abnormal prompt message for other personnel to view, and the second feedback operation is a system response operation.
[0095] It can be seen that the human-computer interaction terminal comprehensively predicts the use status of the smart toilet through the first state information, the second state information, and the sound information, and then interacts with the user by displaying the inquiry information on the inner display screen in an abnormal use state, and then updates the display content of the outer display screen to the first display information or the second display information according to the first feedback operation or the second feedback operation in response to the inquiry information. This is conducive to improving the intelligence and accuracy of the human-computer interaction terminal in processing the use data in the toilet, thereby improving the accuracy of judging the user's use status, and then improving the accuracy of the display content of the inner and outer display screens, and improving the use experience of users in the toilet and people waiting outside the toilet.
[0096] In a possible example, determining a prediction result based on the first state information, the second state information, and the sound information includes: constructing first feature data using a one-hot encoding mechanism for the first state information; constructing second feature data using a multi-hot encoding mechanism for the second state information, and performing feature normalization processing on the second feature data to obtain updated second feature data; constructing third feature data for the type of sound contained in the sound information; constructing a target feature vector based on the first feature data, the updated second feature data, and the third feature data; calling a pre-trained usage status prediction model, using the target feature vector as a model input, to obtain a predicted probability that the usage status is normal; and determining the prediction result based on the predicted probability and a preset probability threshold.
[0097] Specifically, the first state information can be converted into a new binary feature by using a one-hot encoding mechanism to construct the first feature data. For example, the first state information includes two variables, namely, the door opening state and the door closing state. The first feature data constructed by using a one-hot encoding mechanism includes (1, 0) and (0, 1), where (1, 0) represents the door opening state and (0, 1) represents the door closing state. In this way, the first state information can be converted into a digital feature that can be understood by the usage state prediction model, so as to facilitate the subsequent use of the usage state prediction model to determine the prediction result.
[0098] Specifically, the second state information can be converted into a new binary feature by constructing the second feature data using a multi-hot encoding mechanism. For example, the second state information includes several variables, such as seat pressure information, flushing event information, and sanitary film rotation event information. Since the second state information may be interested in at least one variable, the constructed second feature data can be (1, 0, 1), that is, there is seat pressure information and sanitary film rotation event information. Constructing the second feature data in this way can convert the second state information into a digital feature that can be understood by the usage state prediction model, so as to facilitate the subsequent use of the usage state prediction model to determine the prediction result. Furthermore, the second feature data can also be normalized using a normalization method such as MinMaxScaler or StandardScaler to make the second feature data more suitable for the training optimization of the usage state prediction model and improve the accuracy of the prediction result.
[0099] Specifically, the types of sounds contained in the sound information refer to user sound information, mobile phone sound information, etc. For the sound information, the multi-hot encoding mechanism as described above can be used to construct the third feature data, and the third feature data can be normalized, which will not be further elaborated here.
[0100] In a specific implementation, a target feature vector can be further constructed based on the constructed first feature data, second feature data, and third feature data, and the target feature vector can be input into the pre-trained usage status prediction model so that the model outputs the predicted probability that the usage status is normal. Then the preset probability threshold is obtained, and the predicted probability is compared with the preset probability threshold, so as to determine the prediction result according to the comparison result. For example, the preset probability threshold can be set to 80%. If the predicted probability is greater than or equal to the predicted probability threshold, the predicted result is a normal usage status. If the predicted probability is less than the predicted probability threshold, the predicted result is an abnormal usage status.
[0101] It can be seen that in this example, the first state information, the second state information, and the sound information are constructed into the first feature data, the second feature data, and the third feature data through the one-hot encoding mechanism and the multi-hot encoding mechanism respectively, which can make the feature data construction more in line with the characteristics of each information, thereby ensuring the accuracy of the target feature vector constructed by each feature, and then ensuring the reliability of the final prediction result.
[0102] In a possible example, the usage status prediction model is a logistic regression binary classification model. Using a logistic regression binary classification model to determine the prediction result can reduce the possibility of overfitting and improve the efficiency of determining the prediction result.
[0103] In a possible example, the functional formula of the usage status prediction model is as follows: f(d)=1÷(1+e d ), d=W1×X1+W2×X2+W3×X3+W0; wherein f(d) represents the predicted probability that the usage status is normal, X1 is the first feature data, X2 is the updated second feature data, X3 is the third feature data, W1 is the first preset coefficient corresponding to X1, W2 is the second preset coefficient corresponding to X2, W3 is the third preset coefficient corresponding to X3, and W0 is the bias coefficient.
[0104] Among them, d is the undetermined parameter in the function. During training, sample training can be used to make the parameter d have a very accurate prediction of the data in the training set. The output f(d) is mapped between (0, 1). If f(d) is greater than 0.5, the prediction result is determined to be a normal use state. If f(d) is less than or equal to 0.5, the prediction result is determined to be an abnormal use state.
[0105] It can be seen that in this example, through the function formula f(d)=1÷(1+e d ) can concisely and clearly determine the predicted probability that the usage status is normal based on the first feature data, the second feature data, and the third feature data.
[0106] In a possible example, the formula of the loss function LOSS of the usage state prediction model is as follows: Where N is the number of samples in the training set, i represents the i-th sample in N samples, y(i) represents the true probability in the sample, y′(i) represents the predicted probability of the model, λ represents the L2 regularization term coefficient, and Regularization(ω) represents the L2 regularization term.
[0107] It can be seen that in this example, by optimizing the function formula of the usage status prediction model through the loss function, the predicted probability can be made closer to the true probability, thereby improving the accuracy of the prediction results and meeting the actual usage of users.
[0108] In one possible example, when the prediction result is a normal usage status, the method further includes: displaying a usage time setting interface on the inner display screen; determining the target usage time in response to a first setting operation, the first setting operation being a user response operation; and determining the target usage time in response to a second setting operation based on the average usage time obtained by pre-statistical analysis of historical usage time data, the second setting operation being a system response operation.
[0109] Among them, the target usage time refers to the length of time the user expects to use the toilet.
[0110] Specifically, after the user enters the toilet, if the prediction result obtained by the human-computer interaction terminal is a normal use state, the use time setting interface can be displayed on the inner display screen. Then, it is detected whether a user response operation is detected within the preset time. If a user response operation is detected, the user response operation is determined to be a first setting operation, and the user's target use time is determined based on the first setting operation. If no user response operation is detected, it is determined according to the preset program that there is a second setting operation, and the target use time is determined based on the average use time obtained by pre-statistically analyzing the historical use time data according to the second setting operation.
[0111] In a specific implementation, the average usage time obtained by pre-statistical analysis of historical usage time data refers to the time obtained by statistical analysis of the target usage time data set according to the history of the toilet seat, and the time can be any value such as 5 minutes, 10 minutes, 15 minutes, etc. The step of determining the target usage time according to the average usage time obtained by pre-statistical analysis of historical usage time data specifically includes: obtaining a historical usage time data set, determining the average usage time of multiple historical usage times in the historical usage time data set, and determining the average usage time as the target usage time. Among them, the historical usage time data can be the historical usage data of the toilet seat corresponding to it recorded by the human-computer interaction terminal. Alternatively, the state perception system also includes an integrated server, which can interact with the human-computer interaction terminal corresponding to each toilet seat for data exchange, and the historical usage time data set can be multiple historical usage times recorded by multiple human-computer interaction terminals obtained by the integrated server.
[0112] Specifically, before determining the average usage duration of multiple historical usage durations in the historical usage duration data set, multiple historical usage durations that meet the preset threshold range can be screened out in the historical usage data set, and then the average usage duration of the multiple historical usage durations can be determined. This reduces the possibility that the target usage duration set in response to the second setting operation is too short or too long. The preset threshold range can be set according to an empirical value.
[0113] For example, the usage time setting interface displayed on the inner display screen may be as follows: Figure 6 The interface shown is also the fourth interface 60. Figure 6 As shown, the fourth interface 60 includes a second theme description control 610, a duration setting display control 620, a setting method prompt control 630, and an other content prompt control 640. Among them, the second theme description control 610 is used to represent the theme of the current interface. The duration setting display control 620 is used for the user to set and display the usage time by touching. For example, the initial duration displayed by the duration setting display control 620 can be 5 minutes, and the user can increase or decrease the duration according to the step length by touching. The step length of the duration changed by a single touch of the user can be 1 minute or 5 minutes, etc. The setting method prompt control 630 is used to prompt the user how to set the usage time. The other content prompt control 640 is used to add other content to remind the user, for example, Figure 6 "For the convenience of you and others, please set the estimated usage time reasonably."
[0114] It can be seen that in this example, after the user enters the toilet, setting the target usage time in response to the first setting operation or the second setting operation can improve the diversity and reliability of the target usage time setting. At the same time, determining the target usage time based on the average usage time obtained by pre-statistical analysis of historical usage time data can also improve the accuracy and practicality of the target usage time while completing the target usage time setting.
[0115] In one possible example, after determining the target usage time, the method further includes: displaying usage time information on the inner display screen and the outer display screen until the target usage time timing ends, and controlling the smart door lock to perform a door opening operation when the target usage time timing ends, the usage time information including the used time information and the remaining usage time information; or, displaying the usage time information on the inner display screen and the outer display screen until it is detected that a user manually opens the smart door lock.
[0116] Specifically, after the human-computer interaction system determines the target usage time, the usage time information can be displayed on the inner display screen and the outer display screen. The specific presentation form of the usage time information can refer to the following. Figure 4 (c) Interface settings shown.
[0117] In a specific implementation, during the target usage time timing process, the inner display screen and the outer display screen can always display the usage time information. When the target usage time timing ends, the human-computer interaction terminal can send an unlock command to the smart door lock to instruct the smart door lock to perform the door opening operation through the unlock command. Specifically, before the human-computer interaction terminal sends the unlock command to the smart door lock, it can first send an overtime message to the inner display screen to display the overtime inquiry information through the inner display screen. If the user increases the usage time, the human-computer interaction terminal will not send an unlock command to the smart door lock, and will determine the increased usage time as the updated target usage time, and update the usage time information according to the updated target usage time, and display the updated usage time information on the inner display screen and the outer display screen, and then repeat the above timing operation.
[0118] In the specific implementation, during the target usage time timing, the inner display screen and the outer display screen can always display the usage time information. If the human-computer interaction terminal detects that the user manually opens the smart door lock through the smart door lock before the target usage time is counted, the usage time information will stop being displayed and wait for the next user to use it.
[0119] It can be seen that in this example, by displaying the target usage time on the inner display screen, the user in the toilet can be reminded to reasonably control the usage time, and by displaying the target usage time on the outer display screen, other people outside the toilet (such as other waiting users or property management personnel, etc.) can be assisted to know the usage of the toilet and make reasonable arrangements. In addition, when the user is provided with the option to manually open the smart door lock to ensure user use, by controlling the smart door lock to perform the door opening operation when the target usage time timing ends, it is convenient to provide assistance to the user when the user faints in the toilet and ensure the user's safety.
[0120] The embodiments of the present application are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of the present application. At the same time, for general technical personnel in this field, according to the idea of the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.
Claims
1. A control method for intelligently sensing multimodal information, It is characterized in that A human-computer interaction terminal applied to a state perception system, the state perception system comprising the human-computer interaction terminal arranged through the inner and outer sides of a door panel, a smart door lock arranged in a locking area of the door panel, and a smart toilet in a toilet seat corresponding to the door panel, the human-computer interaction terminal being respectively connected to the smart door lock and the smart toilet in communication, the human-computer interaction terminal comprising an inner display screen, an outer display screen, a sound receiving module and a sound broadcasting module, the method comprising: receiving first status information of the door panel reported by the smart door lock, and receiving second status information reported by the smart toilet, wherein the first status information is used to characterize the open state and the closed state of the door panel, and the second status information is used to characterize the seat pressure information, the flushing event information, and the sanitary film rotation event information; The sound receiving module collects sound information of the space in the inner direction of the door panel, wherein the sound information includes at least one of the following: user sound information and mobile phone sound information; Determine a prediction result according to the first state information, the second state information and the sound information, wherein the prediction result is used to characterize the predicted usage state of the smart toilet, and the prediction result includes a normal usage state and an abnormal usage state; When the prediction result is an abnormal usage state, displaying an inquiry message on the inner display screen, wherein the inquiry message is used to inquire the user whether an abnormality occurs; In response to a first feedback operation for the inquiry information, first display information is displayed on the outer display screen, the first display information is used to display an abnormal state message, the abnormal state message is used to characterize an abnormal state, and the first feedback operation is a user response operation; In response to a second feedback operation for the inquiry message, second display information is displayed on the outer display screen, the second display information is used to display an abnormal prompt message for other personnel to view, and the second feedback operation is a system response operation.
2. The method according to claim 1, It is characterized in that The determining the prediction result according to the first state information, the second state information and the sound information includes: Constructing first feature data using a one-hot encoding mechanism for the first state information; constructing second feature data using a multi-hot encoding mechanism for the second state information, and performing feature normalization processing on the second feature data to obtain updated second feature data; Constructing third feature data according to the type of sound contained in the sound information; constructing a target feature vector according to the first feature data, the updated second feature data, and the third feature data; Calling a pre-trained usage status prediction model, taking the target feature vector as a model input, and obtaining a prediction probability that the usage status is normal; The prediction result is determined according to the prediction probability and a preset probability threshold.
3. The method according to claim 2, It is characterized in that The usage status prediction model is a logistic regression binary classification model.
4. The method according to claim 3, It is characterized in that The function formula of the usage status prediction model is as follows: f(d)=1÷(1+e d ), d=W 1 ×X 1 +W 2 ×X 2 +W 3 ×X 3 +W 0 ; Wherein, f(d) represents the predicted probability that the usage status is normal, X 1 is the first feature data, X 2 is the second feature data after update, X 3 is the third characteristic data, W 1 For X 1 The corresponding first preset coefficient, W 2 For X 2 The corresponding second preset coefficient, W 3 For X 3 The corresponding third preset coefficient, W 0 is the bias coefficient.
5. The method according to any one of claims 2 to 4, It is characterized in that The formula of the loss function LOSS of the usage state prediction model is as follows: Where N is the number of samples in the training set, i represents the i-th sample in N samples, y(i) represents the true probability in the sample, y′(i) represents the predicted probability of the model, λ represents the L2 regularization term coefficient, and Regularization(ω) represents the L2 regularization term.
6. The method according to claim 1, It is characterized in that The abnormality includes at least one of equipment abnormality, physical abnormality, and judgment abnormality. The equipment abnormality refers to an abnormality in at least one of the smart door lock and the smart toilet. The judgment abnormality refers to an error in the conclusion that the predicted result is an abnormal usage state.
7. The method according to claim 6, It is characterized in that In response to the first feedback operation for the inquiry information, displaying first display information on the outer display screen includes: If the first feedback operation is to mark the device as abnormal, then the abnormal status message is a device abnormality prompt message; If the first feedback operation is to mark a physical abnormality, the abnormal state message is the abnormal prompt message; If the first feedback operation is marking the judgment as abnormal, then the abnormal status message is a usage duration update message.
8. The method according to claim 1, It is characterized in that When the prediction result is a normal use state, the method further includes: Displaying a usage time setting interface on the inner display screen; In response to a first setting operation, determining a target usage duration, wherein the first setting operation is a user response operation; In response to a second setting operation, the target usage duration is determined according to an average usage duration obtained by pre-statistically analyzing historical usage duration data, and the second setting operation is a system response operation.
9. The method according to claim 8, It is characterized in that After determining the target usage time, the method further includes: Displaying usage time information on the inner display screen and the outer display screen until the target usage time timing ends, and controlling the smart door lock to perform a door opening operation when the target usage time timing ends, the usage time information including the used time information and the remaining usage time information; Alternatively, the usage time information is displayed on the inner display screen and the outer display screen until it is detected that the user manually opens the smart door lock.
10. A state perception system, It is characterized in that The state perception system includes a human-machine interaction terminal arranged on the inner and outer sides of the door panel, a smart door lock arranged in the locking area of the door panel, and a smart toilet in the toilet seat corresponding to the door panel, wherein the human-machine interaction terminal is respectively connected to the smart door lock and the smart toilet for communication, and the human-machine interaction terminal includes an inner display screen, an outer display screen, a sound receiving module and a sound broadcasting module; The smart door lock is used to collect first status information of the corresponding door panel and report the first status information to the human-computer interaction terminal, wherein the first status information is used to represent the open state and the closed state of the door panel; The smart toilet is used to collect the second state information and report the second state information to the human-computer interaction terminal, wherein the second state information is used to represent the seat pressure information, flushing event information, and sanitary film rotation event information; The human-computer interaction terminal is used to receive the first status information of the door panel reported by the smart door lock, and to receive the second status information reported by the smart toilet; The sound receiving module collects sound information of the space in the inner direction of the door panel, wherein the sound information includes at least one of the following: user sound information and mobile phone sound information; Determine a prediction result according to the first state information, the second state information and the sound information, wherein the prediction result is used to characterize the predicted usage state of the smart toilet, and the prediction result includes a normal usage state and an abnormal usage state; When the prediction result is an abnormal usage state, displaying an inquiry message on the inner display screen, wherein the inquiry message is used to inquire the user whether an abnormality occurs; In response to a first feedback operation for the inquiry information, first display information is displayed on the outer display screen, the first display information is used to display an abnormal state message, the abnormal state message is used to characterize an abnormal state, and the first feedback operation is a user response operation; In response to a second feedback operation for the inquiry message, second display information is displayed on the outer display screen, the second display information is used to display an abnormal prompt message for other personnel to view, and the second feedback operation is a system response operation.
Citation Information
Patent Citations
Public washroom management method based on internet of things
CN109361751A
Abnormality prediction method, abnormality prediction device, electronic equipment and storage medium
CN110517768A
Abnormality detection rule generation method and device
CN118101352A
Toilet door lock with call for help function
CN206785122U
Bathroom monitor
JP2004126954A
