Intention deviation prompting method and device based on behavior habit, medium and equipment

Abstract

Embodiments of the present application provide a method for prompting deviation of an intended behavior habit based on a behavior habit, comprising: predicting an intended behavior habit of a user according to an actual environment state generated by an actual behavior habit of the user, using a pre-constructed behavior habit model; predicting a predicted behavior habit of the user according to an obtained current environment state, using the behavior habit model; determining a difference between the predicted behavior habit and the intended behavior habit; and issuing a prompt information when the difference meets a set threshold. Embodiments of the present application also provide an apparatus for prompting deviation of an intended behavior habit based on a behavior habit, a computer readable storage medium and an electronic device.

Description

Technical Field

[0001] This invention relates to the field of smart home technology, and more particularly to a method for inferring user intent in the current environment based on user behavior characteristics and environmental characteristics. More specifically, it relates to a method, device, medium, and equipment for indicating intent deviation based on behavioral habits. Background Technology

[0002] With the development of technologies such as 5G communication, edge computing, cloud storage, and artificial intelligence, the smart home field has entered a new stage of development. The concept of "whole-house intelligence" is gradually becoming popular, and the smart home market is constantly expanding. Remotely connecting home appliances via mobile applications (APPs) has become a standard feature of smart home products. Smart voice assistants are also widely used in smart home systems, undertaking multiple roles such as front-end interaction and back-end control. In some products, this voice assistant serves as the central control console for the entire home system.

[0003] In realizing the concept of this invention, the inventors discovered at least the following problems in related technologies: In smart home scenarios, existing technologies mainly consider that users make requests through user interaction interfaces (e.g., graphical user interface (GUI), voice user interface (VUI), etc.), and the smart agent passively responds to the corresponding needs. For example, users can send commands to smart home devices via voice, such as "set an alarm for 8 a.m. tomorrow," "open the curtains," or "turn up the TV volume." To date, such smart agents have not demonstrated the ability to proactively provide alerts or feedback.

[0004] Therefore, how to enable intelligent agents to detect potential user action or planning errors and proactively provide warnings or feedback is a technical problem that urgently needs to be solved by professionals in this field. Summary of the Invention

[0005] In view of this, the technical problem to be solved by the present invention is to provide a method, device, medium and equipment for intention deviation prompting based on behavioral habits, which solves the problem in the existing smart home field that smart agents cannot detect potential user action errors or planning errors and cannot proactively provide warnings or feedback.

[0006] To address the aforementioned technical problems, specific embodiments of the present invention provide a method for indicating intention deviation based on behavioral habits, comprising: predicting the user's intentional behavioral habits using a pre-built behavioral habit model based on the actual environmental state generated by the user's actual behavioral habits; predicting the user's predicted behavioral habits using the behavioral habit model based on the acquired current environmental state; determining the difference between the predicted behavioral habits and the intentional behavioral habits; and issuing a prompt message when the difference meets a set threshold.

[0007] Optionally, the step of predicting a user's intended behavioral habits based on the actual environmental state generated by the user's actual behavioral habits and using a pre-built behavioral habit model includes: predicting the user's intended environmental state at the current moment based on the actual environmental state generated by the user's actual behavioral habits within a set time window and using the trajectory of environmental state changes generated by behavioral habits in the pre-built behavioral habit model, wherein, in chronological order, the current moment is a later moment in the set time window; and predicting the user's behavioral habits in the intended environmental state using the mapping relationship between the environmental state and behavioral habits in the behavioral habit model, thereby obtaining the intended behavioral habits.

[0008] Optionally, the prediction of the user's intentional environmental state at the current moment is achieved through a search algorithm.

[0009] Optionally, the step of predicting the user's predicted behavioral habits based on the obtained current environmental state and using the behavioral habit model includes: using the mapping relationship between the environmental state and behavioral habits in the behavioral habit model to predict the user's behavioral habits in the current environmental state at the current moment, thereby obtaining the predicted behavioral habits.

[0010] Optionally, the step of pre-constructing a behavioral habit model includes: operation S11: obtaining a first historical environmental state and a first historical behavioral habit under the first historical environmental state; operation S12: training the behavioral habit model based on the first historical environmental state and the first historical behavioral habit to obtain training model parameters; and operation S15: applying the training model parameters as model parameters of the behavioral habit model, wherein the model parameters include the mapping relationship between environmental state and behavioral habit and the trajectory of environmental state changes generated by behavioral habit.

[0011] Optionally, the step of pre-constructing the behavioral habit model before operation S15 further includes: operation S13: obtaining a second historical environment state and a second historical behavioral habit under the second historical environment state; operation S14: verifying the training model parameters based on the second historical environment state and the second historical behavioral habit; operation S15: if the verification is successful, applying the training model parameters as model parameters of the behavioral habit model; and operation S16: if the verification fails, repeating operations S11 to S14 until the verification is successful.

[0012] Optionally, both the first historical environmental state and the second historical environmental state include m environmental dimension data, wherein the m environmental dimension data are at least one of space, temperature, humidity, ambient sound, light, position of objects in space, posture of objects in space, and state of objects in space; and / or both the first historical behavioral habits and the second historical behavioral habits include n behavioral dimension data, wherein the n behavioral dimension data are at least one of the user's facial expression, voice, operation action, and posture.

[0013] Optionally, the step of training the behavior habit model based on the first historical environment state and the first historical behavior habit to obtain training model parameters includes: extracting features from the first historical environment state and the first historical behavior habit to obtain a first historical environment feature sequence and a first historical behavior feature sequence; performing data dimensionality reduction processing on the first historical environment feature sequence and the first historical behavior feature sequence to obtain first historical environment dimensionality-reduced data and first historical behavior dimensionality-reduced data; and training the behavior habit model using the first historical environment dimensionality-reduced data and the first historical behavior dimensionality-reduced data to obtain training model parameters.

[0014] Optionally, verifying the training model parameters based on the second historical environment state and the second historical behavior habit includes: extracting features from the second historical environment state and the second historical behavior habit respectively to obtain a second historical environment feature sequence and a second historical behavior feature sequence; performing data dimensionality reduction processing on the obtained second historical environment feature sequence and the second historical behavior feature sequence respectively to obtain second historical environment dimensionality reduction data and second historical behavior dimensionality reduction data; and using the second historical environment dimensionality reduction data and the second historical behavior dimensionality reduction data to verify the training model parameters.

[0015] Optionally, the m environmental dimension data are time-aligned data; and / or the n behavioral dimension data are time-aligned data.

[0016] Another aspect of this invention provides an intent deviation prompting device based on behavioral habits, comprising: a first prediction module, configured to predict a user's intended behavioral habits based on the actual environmental state generated by the user's actual behavioral habits and using a pre-built behavioral habit model; a second prediction module, configured to predict a user's predicted behavioral habits based on the acquired current environmental state and using the behavioral habit model; a determination module, configured to determine the difference between the predicted behavioral habits and the intended behavioral habits; and a prompting module, configured to issue a prompting message when the difference meets a set threshold.

[0017] Another aspect of the present invention provides an electronic device including one or more processors and a storage device, wherein the storage device is used to store executable instructions, which, when executed by the processor, implement the method of the present invention.

[0018] Another aspect of the present invention provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method of the present invention.

[0019] Another aspect of the present invention provides a computer program, which includes computer-executable instructions that, when executed, implement the method of the present invention.

[0020] In embodiments of this invention, potential user action or planning errors are taken into account. User behavior habits are inferred from two levels: first, based on the currently measured environmental state, a behavior habit model predicts the user's predicted behavior habits, which may contain errors; second, based on actual behavior habits and the resulting environmental state, a behavior habit model predicts the user's intended behavior habits, which are more consistent with the user's behavior habits and logic than the predicted ones. When a discrepancy arises between these two levels of behavior habits, i.e., when the difference meets a set threshold, a prompt is issued. This enables the intelligent agent used in the method of this invention to detect potential user action or planning errors and proactively provide warnings or feedback.

[0021] It should be understood that the above general description and the following specific embodiments are merely exemplary and illustrative, and do not limit the scope of the invention. Attached Figure Description

[0022] The accompanying drawings, which are part of the specification of this invention, illustrate exemplary embodiments of the invention. The drawings, together with the description in the specification, serve to illustrate the principles of the invention.

[0023] Figure 1 This is a schematic flowchart illustrating a method for indicating intention deviation based on behavioral habits, provided as a specific embodiment of the present invention.

[0024] Figure 2 This is a schematic flowchart illustrating how a user's intentional behavioral habits can be predicted based on the actual environmental state generated by the user's actual behavioral habits and using a pre-built behavioral habit model, as provided in a specific embodiment of the present invention.

[0025] Figure 3 This is a schematic diagram of an environmental state change trajectory caused by behavioral habits, provided as a specific embodiment of the present invention.

[0026] Figure 4 This is a schematic diagram illustrating a predicted trajectory of environmental state changes caused by behavioral habits, provided as a specific embodiment of the present invention.

[0027] Figure 5 This is a schematic flowchart illustrating how a user's predictive behavioral habits are predicted based on the acquired current environmental state and using a behavioral habit model, as provided in a specific embodiment of the present invention.

[0028] Figure 6 This is a schematic flowchart illustrating a pre-built behavioral habit model provided for a specific embodiment of the present invention.

[0029] Figure 7 This is a schematic flowchart illustrating another pre-built behavioral habit model provided in a specific embodiment of the present invention.

[0030] Figure 8 This is a schematic flowchart illustrating how to train a behavioral habit model based on a first historical environmental state and a first historical behavioral habit, and obtain the training model parameters, as provided in a specific embodiment of the present invention.

[0031] Figure 9 This is a schematic flowchart illustrating how to verify the parameters of a training model based on a second historical environmental state and second historical behavioral habits, as provided in a specific embodiment of the present invention.

[0032] Figure 10 This is a structural block diagram of an intent deviation prompting device based on behavioral habits, provided for a specific embodiment of the present invention. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the spirit of the contents disclosed in the present invention will be clearly explained below with reference to the accompanying drawings and detailed description. After understanding the embodiments of the present invention, any person skilled in the art can make changes and modifications based on the technology taught in the present invention without departing from the spirit and scope of the present invention.

[0034] The illustrative embodiments and descriptions of the present invention are used to explain the invention, but are not intended to limit the invention. Furthermore, elements / components using the same or similar reference numerals in the drawings and embodiments are used to represent the same or similar parts.

[0035] The terms "first," "second," etc., used in this document are not intended to specifically refer to order or sequence, nor are they intended to limit the invention. They are merely used to distinguish elements or operations described using the same technical terms.

[0036] The directional terms used in this article, such as up, down, left, right, front, or back, are for reference only when referring to the accompanying drawings. Therefore, the use of directional terms is for illustrative purposes and not to limit this work.

[0037] The terms “include,” “including,” “have,” “contain,” etc., used in this article are all open-ended terms, meaning that they include but are not limited to.

[0038] The term "and / or" as used herein includes any or all of the things mentioned.

[0039] The term "multiple" in this article includes "two" and "more than two"; the term "multiple groups" in this article includes "two groups" and "more than two groups".

[0040] The terms "approximately," "about," etc., used herein are intended to modify any quantity or error that may vary slightly, but these slight variations or errors do not change the essence of the quantity or error. Generally, the range of slight variations or errors modified by such terms may be 20% in some embodiments, 10% in others, 5% in still others, or other values. Those skilled in the art should understand that the aforementioned values ​​can be adjusted according to actual needs and are not limited thereto.

[0041] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0042] When expressions such as "at least one of A, B, and C" are used, they should generally be interpreted in accordance with the meaning commonly understood by those skilled in the art (e.g., "a system having at least one of A, B, and C" should include, but is not limited to, systems having A alone, having B alone, having C alone, having A and B, having A and C, having B and C, and / or having A, B, and C, etc.). When expressions such as "at least one of A, B, or C" are used, they should generally be interpreted in accordance with the meaning commonly understood by those skilled in the art (e.g., "a system having at least one of A, B, or C" should include, but is not limited to, systems having A alone, having B alone, having C alone, having A and B, having A and C, having B and C, and / or having A, B, and C, etc.). Those skilled in the art should also understand that any conjunction and / or phrase that substantially arbitrarily indicates two or more optional items, whether in the specification, claims, or drawings, should be understood to indicate the possibility of including one of these items, either of these items, or both items. For example, the phrase “A or B” should be understood as including the possibility of “A” or “B”, or “A and B”.

[0043] Figure 1 This is a schematic flowchart illustrating a method for indicating intention deviation based on behavioral habits, provided as a specific embodiment of the present invention.

[0044] like Figure 1 As shown, the intention deviation prompting method based on behavioral habits may include the following operations S101 to S104.

[0045] In operation S101: Based on the actual environmental state generated by the user's actual behavioral habits, the user's intentional behavioral habits are predicted using a pre-built behavioral habit model.

[0046] In embodiments of the present invention, a user's actual behavioral habits may include at least one of the user's facial expressions, voice, operational actions, and posture. Specifically, facial expressions can be captured by a camera array, voice can be captured by a microphone, operational actions can be captured by user operation signals, and posture can be captured by an inertial measurement unit. Of course, a user's actual behavioral habits may also include other behavioral habits; these are merely illustrative examples and should not be construed as limiting the present invention.

[0047] In embodiments of the present invention, the actual environmental state may include at least one of space, temperature, humidity, ambient sound, light, the position of objects within the space, the orientation of objects within the space, and the state of objects within the space. Specifically, at least one of space, temperature, humidity, ambient sound, light, the position of objects within the space, the orientation of objects within the space, and the state of objects within the space can be collected by means of RGB cameras, infrared cameras, microphones, inertial measurement units, temperature and humidity sensors, light sensors, mobile phone operation signals, and PC operation signals. Of course, the actual environmental state may also include other environmental states; this is merely an example and should not be construed as a limitation of the present invention.

[0048] It should be noted that the actual environmental state is generated by the user's actual behavior and habits. For example, when the user performs an operation on the humidifier and adjusts the humidity in the bedroom to value 'a', the actual environmental state generated by the user's actual behavior and habits is that the humidifier is on and the humidity in the bedroom is value 'a'.

[0049] Secondly, in operation S102: based on the current environmental state obtained, the user's predictive behavioral habits are predicted using a behavioral habit model.

[0050] It is understood that the current environmental state is measured at the current moment and can be at least one of the following: space, temperature, humidity, ambient sound, light, the position of objects within the space, the posture of objects within the space, and the state of objects within the space. Specifically, it can be measured by an RGB camera, an infrared camera, a microphone, an inertial measurement unit, a temperature and humidity sensor, a light sensor, mobile phone operation signals, and PC operation signals, etc. Of course, the current environmental state can also include other environmental states; this is only an example and should not be construed as a limitation of the present invention.

[0051] Next, in operation S103: determine the difference between predicted behavioral habits and intended behavioral habits.

[0052] Then, in operation S104: when the difference meets the set threshold, a prompt message is issued.

[0053] In embodiments of the present invention, the threshold can be a numerical value. When the difference meets the threshold, it can be understood that the difference is greater than or equal to the numerical value, and a prompt message can be issued at this time.

[0054] In embodiments of the present invention, the set threshold can be a numerical range. When the difference meets the set threshold, it can be understood that the difference is within the numerical range, and a prompt message can be issued at this time.

[0055] In embodiments of the present invention, the prompting information may include reminders given to the user through voice interaction or text interaction.

[0056] In embodiments of this invention, potential user action or planning errors are taken into account. User behavior habits are inferred from two levels: first, based on the currently measured environmental state, a behavior habit model predicts the user's predicted behavior habits, which may contain errors; second, based on actual behavior habits and the resulting environmental state, a behavior habit model predicts the user's intended behavior habits, which are more consistent with the user's behavior habits and logic than the predicted ones. When a discrepancy arises between these two levels of behavior habits, i.e., when the difference meets a set threshold, a prompt is issued. This enables the intelligent agent used in the method of this invention to detect potential user action or planning errors and proactively provide warnings or feedback.

[0057] Figure 2 This is a schematic flowchart illustrating how a user's intentional behavioral habits can be predicted based on the actual environmental state generated by the user's actual behavioral habits and using a pre-built behavioral habit model, as provided in a specific embodiment of the present invention.

[0058] like Figure 2 As shown, operation S101 predicts the user's intentional behavior habits based on the actual environmental state generated by the user's actual behavior habits and using a pre-built behavior habit model. This may include the following operations S1011 to S1012.

[0059] In operation S1011: Based on the actual environmental state generated by the user's actual behavior habits within the set time window, the user's intended environmental state at the current moment is predicted by using the environmental state change trajectory generated by the behavior habits of the pre-built behavior habit model. In terms of time sequence, the current moment is the later moment of the set time window.

[0060] Next, in operation S1012: using the mapping relationship between the environmental state and behavioral habits of the behavioral habit model, predict the user's behavioral habits under the intentional environmental state, and obtain the intentional behavioral habits.

[0061] In embodiments of the present invention, the time window can be a short time segment, such as several seconds, several minutes, or several hours. The time window can be specifically set according to the actual situation, and no specific limitation is made on the time window here.

[0062] In embodiments of the present invention, when a behavioral habit model is pre-constructed, the environmental state change trajectory generated by the behavioral habits in the model can be trained. The environmental state change trajectory generated by the behavioral habits in the model can be as follows: Figure 3 As shown, Figure 3The hollow circles in the diagram represent environmental state feature vectors, and the lines represent the trajectory of environmental state changes caused by behavioral habits. Based on actual behavioral habits and actual environmental states, and according to the trajectory of environmental state changes caused by behavioral habits, a predicted trajectory of environmental state changes caused by the user's behavioral habits can be obtained.

[0063] The predicted trajectory of environmental state changes caused by the user's behavioral habits is as follows: Figure 4 As shown, Figure 4 The hollow circles in the diagram represent environmental state feature vectors, the thin lines represent the trajectory of environmental state changes caused by behavioral habits, and the thick lines represent the predicted trajectory of environmental state changes caused by behavioral habits. This allows us to obtain the user's intended environmental state at the current moment. By utilizing the mapping relationship between environmental states and behavioral habits in the behavioral habit model, we can predict the user's behavioral habits under the intended environmental state, thus obtaining the intended behavioral habits.

[0064] In embodiments of the present invention, for example, data dimensionality reduction processing can be performed on actual behavioral habits and actual environmental states to obtain dimensionality-reduced data of actual behavioral habits and actual environmental states; feature extraction is performed on the dimensionality-reduced data of actual behavioral habits and actual environmental states to obtain feature sequences of actual behavioral habits and actual environmental states; a search algorithm calculates the feature sequences of actual behavioral habits and actual environmental states to obtain the predicted trajectory of the user's environmental state changes caused by behavioral habits, and obtains the user's intentional environmental state feature sequence at the current moment; the intentional environmental state feature sequence is compared with the mapping relationship between environmental states and behavioral habits in the behavioral habit model to obtain the user's behavioral habits at the current moment under the current environmental state, that is, the predicted behavioral habits.

[0065] In embodiments of the present invention, by operating S1011 to S1012, it is easy to predict the user's intentional behavior habits based on the actual environmental state generated by the user's actual behavior habits and using a pre-built behavior habit model.

[0066] In some examples of this invention, the prediction of the user's intent environment state at the current moment is achieved through a search algorithm. This search algorithm may include, but is not limited to, the A* algorithm and heuristic algorithms, etc. Using a search algorithm facilitates the prediction of the user's intent environment state at the current moment.

[0067] Figure 5 This is a schematic flowchart illustrating how a user's predictive behavioral habits are predicted based on the acquired current environmental state and using a behavioral habit model, as provided in a specific embodiment of the present invention.

[0068] like Figure 5As shown, operation S102, based on the acquired current environmental state and using a behavioral habit model, predicts the user's predicted behavioral habits, which may include the following operation S1021.

[0069] In operation S1021: using the mapping relationship between the environmental state and behavioral habits in the behavioral habit model, predict the user's behavioral habits in the current environmental state at the current moment, and obtain the predicted behavioral habits.

[0070] In embodiments of the present invention, for example, the current environmental state can be subjected to data dimensionality reduction processing to obtain environmental dimensionality-reduced data; features can be extracted from the environmental dimensionality-reduced data to obtain an environmental feature sequence; the environmental feature sequence is compared with the mapping relationship between environmental state and behavioral habits in the behavioral habit model to obtain the user's behavioral habits under the current environmental state at the current moment, that is, the predicted behavioral habits. By operating S1021, it is easy to realize the prediction of the user's predicted behavioral habits based on the obtained current environmental state and using the behavioral habit model.

[0071] Figure 6 This is a schematic flowchart illustrating a pre-built behavioral habit model provided for a specific embodiment of the present invention.

[0072] like Figure 6 As shown, pre-constructing a behavioral habit model can include the following operations S11, S12 and S15.

[0073] In operation S11: Obtain the first historical environment state and the first historical behavior habit under the first historical environment state.

[0074] In embodiments of the present invention, the first historical environmental state may include at least one of space, temperature, humidity, ambient sound, light, the position of an object within the space, the posture of an object within the space, and the state of an object within the space. Specifically, at least one of space, temperature, humidity, ambient sound, light, the position of an object within the space, the posture of an object within the space, and the state of an object within the space can be collected by means of an RGB camera, an infrared camera, a microphone, an inertial measurement unit, a temperature and humidity sensor, a light sensor, mobile phone operation signals, and PC operation signals. Of course, the first historical environmental state may also include other environmental states; this is merely an example and should not be construed as a limitation of the present invention.

[0075] In embodiments of the present invention, the first historical behavioral habit may include at least one of the user's facial expressions, voice, operational actions, and posture. Specifically, the user's facial expressions can be collected by a camera array, the user's voice can be collected by a microphone, the user's operational actions can be collected by user operation signals, and the user's posture can be collected by an inertial measurement unit. Of course, the first historical behavioral habit may also include other behavioral habits; this is merely an example and should not be construed as a limitation of the present invention.

[0076] In operation S12: Train the behavior habit model based on the first historical environment state and the first historical behavior habit to obtain the training model parameters.

[0077] In operation S15: the trained model parameters are used as model parameters for the behavior habit model. The model parameters include the mapping relationship between environmental state and behavior habit, as well as the trajectory of environmental state changes caused by behavior habit.

[0078] In embodiments of the present invention, pre-construction of behavioral habit models can be facilitated by operating S11, S12 and S15.

[0079] Figure 7 This is a schematic flowchart illustrating a pre-built behavioral habit model provided for a specific embodiment of the present invention.

[0080] like Figure 7 As shown, pre-constructing a behavioral habit model can include the following operations S11 to S16.

[0081] In operation S11: Obtain the first historical environment state and the first historical behavior habit under the first historical environment state.

[0082] In embodiments of the present invention, the first historical environmental state may include at least one of space, temperature, humidity, ambient sound, light, the position of an object within the space, the posture of an object within the space, and the state of an object within the space. Specifically, at least one of space, temperature, humidity, ambient sound, light, the position of an object within the space, the posture of an object within the space, and the state of an object within the space can be collected by means of an RGB camera, an infrared camera, a microphone, an inertial measurement unit, a temperature and humidity sensor, a light sensor, mobile phone operation signals, and PC operation signals. Of course, the first historical environmental state may also include other environmental states; this is merely an example and should not be construed as a limitation of the present invention.

[0083] In embodiments of the present invention, the first historical behavioral habit may include at least one of the user's facial expressions, voice, operational actions, and posture. Specifically, the user's facial expressions can be collected by a camera array, the user's voice can be collected by a microphone, the user's operational actions can be collected by user operation signals, and the user's posture can be collected by an inertial measurement unit. Of course, the first historical behavioral habit may also include other behavioral habits; this is merely an example and should not be construed as a limitation of the present invention.

[0084] In operation S12: Train the behavior habit model based on the first historical environment state and the first historical behavior habit to obtain the training model parameters.

[0085] In operation S13: Obtain the second historical environment state and the second historical behavior habit under the second historical environment state.

[0086] In embodiments of the present invention, the second historical environmental state may include at least one of space, temperature, humidity, ambient sound, light, the position of an object within the space, the posture of an object within the space, and the state of an object within the space. Specifically, at least one of space, temperature, humidity, ambient sound, light, the position of an object within the space, the posture of an object within the space, and the state of an object within the space can be collected by means of an RGB camera, an infrared camera, a microphone, an inertial measurement unit, a temperature and humidity sensor, a light sensor, mobile phone operation signals, and PC operation signals. Of course, the second historical environmental state may also include other environmental states; this is merely an example and should not be construed as a limitation of the present invention.

[0087] In embodiments of the present invention, the second historical behavioral habit may include at least one of the user's facial expressions, voice, operational actions, and posture. Specifically, the user's facial expressions can be collected by a camera array, the user's voice can be collected by a microphone, the user's operational actions can be collected by user operation signals, and the user's posture can be collected by an inertial measurement unit. Of course, the second historical behavioral habit may also include other behavioral habits; this is merely an example and should not be construed as a limitation of the present invention.

[0088] In operation S14: Validate the parameters of the trained model based on the second historical environment state and the second historical behavioral habits.

[0089] In operation S15: If the verification is successful, the trained model parameters will be used as the model parameters of the behavior habit model.

[0090] In operation S16: If the verification fails, repeat operations S11 to S14 until the verification passes.

[0091] In embodiments of the present invention, a predicted value can be obtained by verifying the behavior habit model using a second historical environmental state and a second historical behavior habit. The labeled data of the second historical environmental state and the second historical behavior habit are the actual values. The loss value between the predicted value and the actual value is calculated using a loss function. If the loss value meets the set model threshold, the verification is considered successful. The training model parameters are then used as the model parameters of the behavior habit model. If the loss value does not meet the set model threshold, the verification is considered unsuccessful. Operations S11 to S14 are repeated until the verification is successful.

[0092] In the embodiments of the present invention, applying the validated training model parameters can make the model parameters of the behavior habit model more accurate, thereby improving the predictive ability of the behavior habit model and making the prediction results more accurate. Operations S11 to S16 facilitate the pre-construction of the behavior habit model.

[0093] According to some embodiments of the present invention, both the first historical environmental state and the second historical environmental state may include m environmental dimension data, wherein the m environmental dimension data are at least one of space, temperature, humidity, ambient sound, light, position of objects in space, posture of objects in space, and state of objects in space; and / or both the first historical behavioral habits and the second historical behavioral habits may include n behavioral dimension data, wherein the n behavioral dimension data are at least one of the user's facial expression, voice, operation action, and posture.

[0094] Therefore, the intention deviation prompting method based on behavioral habits of the present invention can be applied to situations where the environmental state is at least one of space, temperature, humidity, ambient sound, light, the position of an object in the space, the posture of an object in the space, and the state of an object in the space. The intention deviation prompting method based on behavioral habits of the present invention can also be applied to situations where the behavioral habit is at least one of the user's facial expression, voice, operational actions, and posture. Of course, the application scenarios of the intention deviation prompting method based on behavioral habits can also include other scenarios; these are merely illustrative examples and should not be construed as limiting the present invention.

[0095] Figure 8 This is a schematic flowchart illustrating how to train a behavioral habit model based on a first historical environmental state and a first historical behavioral habit, and obtain the training model parameters, as provided in a specific embodiment of the present invention.

[0096] like Figure 8 As shown, operation S12 trains the behavior habit model based on the first historical environment state and the first historical behavior habit, and the training model parameters can include the following operations S121 to S123.

[0097] In operation S121: feature extraction is performed on the first historical environment state and the first historical behavior habit respectively to obtain the first historical environment feature sequence and the first historical behavior feature sequence.

[0098] Then, in operation S122: perform data dimensionality reduction processing on the first historical environment feature sequence and the first historical behavior feature sequence respectively to obtain the first historical environment dimensionality reduction data and the first historical behavior dimensionality reduction data.

[0099] In operation S123: Use the first historical environment dimensionality reduction data and the first historical behavior dimensionality reduction data to train the behavior habit model and obtain the training model parameters.

[0100] In embodiments of the present invention, the primary sources of the first historical behavioral habit data are camera arrays, microphones, inertial measurement units, and user operation signals. After the collected data is aligned, it is fed into the corresponding feature extraction modules for encoding. The feature extraction modules can be implemented using neural networks. For example, data collected by cameras can be extracted using convolutional neural networks (CNNs) to achieve feature extraction for a single frame or within a specified time window; data collected by microphones can be extracted using structures such as recurrent neural networks (RNNs); data collected by inertial measurement units can have features extracted using convolutional neural networks, or traditional methods such as support vector machines and random forests can be used for classification, and the classification results can be one-hot encoded.

[0101] The first historical environmental state data is similar to the first historical behavioral habit data. Besides cameras and microphones, it also includes various embedded sensors to collect state information about the environment and objects within it. The processing method is also similar to that of the first historical behavioral habit data; different feature extraction modules correspond to different sources of raw data, which will not be elaborated upon here.

[0102] After extracting behavioral habit features, the multi-frame, multi-dimensional first historical behavioral habit data constitutes a behavioral feature sequence. For a user's behavioral feature sequence, a behavioral habit model can be generated using data dimensionality reduction methods. Specifically, a self-supervised learning method can be used to learn the user's behavior model, using each behavioral feature sequence as a training slice. The resulting neural network model can predict the behavioral feature sequence of the next frame, or predict the user's possible actions, based on the current and previous behavioral feature sequences. Deep neural networks can be trained using this method; specifically, the network structure can be a Long Short-Term Memory (LSTM) network.

[0103] The process of building user behavior habits is as follows Figure 3 As shown, the first historical environmental state data collected by this invention is mapped to a high-dimensional feature space. Figure 3 Each hollow circle in the diagram represents a feature vector corresponding to a first historical environmental state data observation. Figure 3A line trajectory in the data represents the environmental state transition caused by a user's behavior. A user's behavior will cause the environmental state to shift from one feature point to another; a user's behavioral feature sequence corresponds to a transition trajectory in the environmental state feature space. The process of constructing a user behavior habit model is essentially a dimensionality reduction process for several transition trajectories, which can be achieved through methods such as principal component analysis and neural networks.

[0104] In embodiments of the present invention, by operating S121 to S123, it is convenient to train a behavioral habit model based on a first historical environmental state and a first historical behavioral habit, and obtain training model parameters.

[0105] Figure 9 This is a schematic flowchart illustrating how to verify the parameters of a training model based on a second historical environmental state and second historical behavioral habits, as provided in a specific embodiment of the present invention.

[0106] like Figure 9 As shown, operation S14, which verifies the parameters of the training model based on the second historical environment state and the second historical behavioral habits, may include the following operations S141 to S143.

[0107] In operation S141: feature extraction is performed on the second historical environment state and the second historical behavior habit respectively to obtain the second historical environment feature sequence and the second historical behavior feature sequence.

[0108] Then, in operation S142: perform data dimensionality reduction processing on the obtained second historical environment feature sequence and second historical behavior feature sequence to obtain second historical environment dimensionality reduction data and second historical behavior dimensionality reduction data.

[0109] Next, in operation S143: use the second historical environment dimensionality reduction data and the second historical behavior dimensionality reduction data to verify the trained model parameters.

[0110] In embodiments of the present invention, the second historical behavioral habit data mainly originates from camera arrays, microphones, inertial measurement units, and user operation signals. After the collected data is aligned, it is sent to the corresponding feature extraction modules for encoding. The feature extraction modules can be implemented using neural networks. For example, data collected by cameras can be used to extract features from a single frame or within a specified time window using a convolutional neural network (CNN); data collected by microphones can be used to extract features using a structure such as a recurrent neural network (RNN); data collected by the inertial measurement unit can be used to extract features using convolutional neural networks, or traditional methods such as support vector machines and random forests can be used for classification, and the classification results can be one-hot encoded.

[0111] The second historical environmental state data is similar to the second historical behavioral habit data. Besides cameras and microphones, it also includes various embedded sensors to collect state information about the environment and objects within it. The processing method is also similar to that of the second historical behavioral habit data; different feature extraction modules correspond to different sources of raw data, which will not be elaborated upon here.

[0112] After extracting behavioral habit features, the multi-frame, multi-dimensional second historical behavioral habit data constitutes a behavioral feature sequence. For a user's behavioral feature sequence, a user behavioral habit model can be generated using data dimensionality reduction methods. Specifically, a self-supervised learning method can be used to learn the user's behavior model, using each behavioral feature sequence as a test slice to verify the obtained neural network model, i.e., the training model parameters of the behavioral habit model.

[0113] In embodiments of the present invention, by operating S141 to S143, it is convenient to verify the training model parameters based on the second historical environmental state and the second historical behavioral habits.

[0114] According to some embodiments of the present invention, the m environmental dimension data are time-aligned data; and / or the n behavioral dimension data are time-aligned data. In other words, the m environmental dimension data and / or the n behavioral dimension data can be aligned according to a unified time.

[0115] In embodiments of the present invention, time alignment is performed at the software level, such as by connecting the involved sensors to the same local area network, and having a central control unit broadcast clock signals or broadcast time data, which are received by each sensor and added to the data packet as tags.

[0116] In embodiments of the present invention, time alignment is performed at the hardware level, such as using a uniform electrical signal to trigger sensor acquisition, so that all sensors start and stop sampling at the same time.

[0117] Figure 10 This is a structural block diagram of an intent deviation prompting device based on behavioral habits, provided for a specific embodiment of the present invention.

[0118] like Figure 10 As shown, the intention deviation prompting device 10 based on behavioral habits may include a first prediction module 1, a second prediction module 2, a determination module 3, and a prompting module 4.

[0119] Specifically, the first prediction module 1 is used to perform operation S101: predict the user's intended behavioral habits based on the actual environmental state generated by the user's actual behavioral habits and using a pre-built behavioral habit model; the second prediction module 2 is used to perform operation S102: predict the user's predicted behavioral habits based on the acquired current environmental state and using the behavioral habit model; the determination module 3 is used to perform operation S103: determine the difference between the predicted behavioral habits and the intended behavioral habits; and the prompting module 4 is used to perform operation S104: issue a prompt message when the difference meets a set threshold.

[0120] In embodiments of this invention, potential user action or planning errors are taken into account. User behavior habits are inferred from two levels: first, based on the currently measured environmental state, a behavior habit model predicts the user's predicted behavior habits, which may contain errors; second, based on actual behavior habits and the resulting environmental state, a behavior habit model predicts the user's intended behavior habits, which are more consistent with the user's behavior habits and logic than the predicted ones. When a discrepancy arises between these two levels of behavior habits, i.e., when the difference meets a set threshold, a prompt is issued. This enables the intelligent agent used in the method of this invention to detect potential user action or planning errors and proactively provide warnings or feedback.

[0121] The present invention also provides an electronic device comprising: one or more processors; and one or more memories for storing executable instructions that, when executed by the processor, implement the methods described above. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0122] The device includes a computing unit that can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) or loaded from a storage unit into random access memory (RAM). The RAM can also store various programs and data required for device operation. The computing unit, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.

[0123] Multiple components in the device are connected to the I / O interface, including: input units such as keyboards and mice; output units such as various types of displays and speakers; storage units such as disks and optical discs; and communication units such as network interface cards (NICs), modems, and wireless transceivers. The communication unit allows the device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0124] Computing units can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Examples of computing units include, but are not limited to, central processing units (CPUs), graphics processing units (GPUs), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any suitable processors, controllers, microcontrollers, etc. Computing units perform the various methods and processes described above, such as image recognition methods. For example, in some embodiments, image recognition methods may be implemented as computer software programs tangibly contained in a machine-readable medium, such as a storage unit.

[0125] In some embodiments, part or all of the computer program may be loaded and / or installed on the device via ROM and / or a communication unit. When the computer program is loaded into RAM and executed by the computing unit, one or more steps of the image recognition method described above may be performed. Alternatively, in other embodiments, the computing unit may be configured to perform the image recognition method by any other suitable means (e.g., by means of firmware).

[0126] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0127] According to embodiments of the present invention, the method flow according to embodiments of the present invention can be implemented as a computer software program. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the method shown in the flowchart. According to embodiments of the present invention, the electronic devices, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0128] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the present invention.

[0129] According to embodiments of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0130] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0131] Those skilled in the art will understand that the features described in the various embodiments and / or claims of the present invention can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, the features described in the various embodiments and / or claims of the present invention can be combined or combined in various ways without departing from the spirit and teachings of the present invention. All such combinations and / or combinations fall within the scope of the present invention.

[0132] The embodiments of the present invention have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of the invention. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of the invention is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of the invention, and all such substitutions and modifications should fall within the scope of the invention.

Claims

1. A method for indicating intention deviation based on behavioral habits, characterized in that, The method includes: Based on the actual environmental state generated by the user's actual behavioral habits, the user's intentional behavioral habits are predicted using a pre-built behavioral habit model; Based on the current environmental state, the user's predictive behavioral habits are predicted using the behavioral habit model. Determine the difference between the predicted behavioral habit and the intended behavioral habit; and When the difference meets the set threshold, a prompt message is issued. The steps involved in predicting a user's intended behavioral habits based on the actual environmental state generated by the user's actual behavioral habits and using a pre-built behavioral habit model include: Based on the actual environmental state generated by the user's actual behavioral habits within a set time window, and utilizing the trajectory of environmental state changes generated by behavioral habits from a pre-built behavioral habit model, the user's intended environmental state at the current moment is predicted, wherein, in chronological order, the current moment is a later moment within the set time window; and By utilizing the mapping relationship between the environmental state and behavioral habits in the aforementioned behavioral habit model, the user's behavioral habits under the intended environmental state are predicted, thus obtaining the intended behavioral habits. The step of predicting the user's predicted behavioral habits based on the obtained current environmental state and using the behavioral habit model includes: By utilizing the mapping relationship between the environmental state and behavioral habits in the behavioral habit model, the user's behavioral habits in the current environmental state at the current moment are predicted, thus obtaining the predicted behavioral habits.

2. The intention deviation prompting method based on behavioral habits according to claim 1, characterized in that, The prediction of the user's intention environment state at the current moment is achieved through a search algorithm.

3. The intention deviation prompting method based on behavioral habits according to claim 1, characterized in that, The steps for pre-building a behavioral habit model include: Operation S11: Obtain the first historical environment state and the first historical behavior habit under the first historical environment state; Operation S12: Train the behavior habit model based on the first historical environment state and the first historical behavior habit to obtain the training model parameters; and Operation S15: Apply the trained model parameters as model parameters of the behavior habit model, wherein the model parameters include the mapping relationship between environmental state and behavior habit and the trajectory of environmental state change generated by behavior habit.

4. The intention deviation prompting method based on behavioral habits according to claim 3, characterized in that, Prior to operation S15, the step of pre-constructing a behavioral habit model further includes: Operation S13: Obtain the second historical environment state and the second historical behavioral habits under the second historical environment state; Operation S14: Verify the parameters of the training model based on the second historical environment state and the second historical behavior habit; Operation S15: If the verification passes, the trained model parameters are applied as the model parameters of the behavioral habit model; and Operation S16: If the verification fails, repeat operations S11 to S14 until the verification passes.

5. The intention deviation prompting method based on behavioral habits according to claim 4, characterized in that, Both the first historical environmental state and the second historical environmental state include m environmental dimension data, wherein the m environmental dimension data are at least one of space, temperature, humidity, ambient sound, light, position of objects in space, posture of objects in space, and state of objects in space; and / or Both the first historical behavior habit and the second historical behavior habit include n behavioral dimension data, wherein the n behavioral dimension data are at least one of the user's facial expressions, voice, operation actions and postures.

6. The intention deviation prompting method based on behavioral habits according to claim 5, characterized in that, The step of training the behavior habit model based on the first historical environmental state and the first historical behavior habit to obtain the training model parameters includes: Feature extraction is performed on the first historical environment state and the first historical behavior habit respectively to obtain the first historical environment feature sequence and the first historical behavior feature sequence; The first historical environment feature sequence and the first historical behavior feature sequence are subjected to dimensionality reduction processing respectively to obtain the first historical environment dimensionality-reduced data and the first historical behavior dimensionality-reduced data; and The behavioral habit model is trained using the first historical environment dimensionality reduction data and the first historical behavior dimensionality reduction data to obtain the training model parameters.

7. The intention deviation prompting method based on behavioral habits according to claim 5, characterized in that, The step of verifying the training model parameters based on the second historical environment state and the second historical behavioral habits includes: Feature extraction is performed on the second historical environment state and the second historical behavior habit respectively to obtain the second historical environment feature sequence and the second historical behavior feature sequence; The obtained second historical environment feature sequence and the second historical behavior feature sequence are subjected to data dimensionality reduction processing to obtain second historical environment dimensionality-reduced data and second historical behavior dimensionality-reduced data; and The parameters of the trained model are verified using the second historical environment dimensionality reduction data and the second historical behavior dimensionality reduction data.

8. The intention deviation prompting method based on behavioral habits according to claim 5, characterized in that, The m environmental dimension data are time-aligned data; and / or the n behavioral dimension data are time-aligned data.

9. A device for indicating intention deviation based on behavioral habits, characterized in that, The device includes: The first prediction module is used to predict the user's intentional behavior habits based on the actual environmental state generated by the user's actual behavior habits and using a pre-built behavior habit model. The second prediction module is used to predict the user's predictive behavior habits based on the acquired current environmental state and the behavior habit model. The determining module is configured to determine the difference between the predicted behavioral habit and the intended behavioral habit; and The prompting module is used to issue a prompt message when the difference meets a set threshold. The steps involved in predicting a user's intended behavioral habits based on the actual environmental state generated by the user's actual behavioral habits and using a pre-built behavioral habit model include: Based on the actual environmental state generated by the user's actual behavioral habits within a set time window, and utilizing the trajectory of environmental state changes generated by behavioral habits from a pre-built behavioral habit model, the user's intended environmental state at the current moment is predicted, wherein, in chronological order, the current moment is a later moment within the set time window; and By utilizing the mapping relationship between the environmental state and behavioral habits in the aforementioned behavioral habit model, the user's behavioral habits under the intended environmental state are predicted, thus obtaining the intended behavioral habits. The step of predicting the user's predicted behavioral habits based on the obtained current environmental state and using the behavioral habit model includes: By utilizing the mapping relationship between the environmental state and behavioral habits in the behavioral habit model, the user's behavioral habits in the current environmental state at the current moment are predicted, thus obtaining the predicted behavioral habits.

10. An electronic device, characterized in that, include: One or more processors; One or more memories are provided for storing executable instructions that, when executed by the processor, implement the method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The storage medium stores executable instructions that, when executed by a processor, implement the method according to any one of claims 1 to 8.

Images