Device state control method and apparatus, storage medium, and electronic device

Abstract

The application discloses a device state control method and device, a storage medium and an electronic device. The device state control method comprises the following steps: acquiring current regional perception data collected by a group of perception sensors in a target region at a current time; analyzing a current object behavior feature and a current regional environment feature from the current regional perception data to obtain a regional perception feature at the current time, and fusing the object behavior feature and the regional environment feature at the same time to obtain the regional perception feature; predicting a regional perception feature at a next time according to a regional perception feature sequence, wherein the regional perception feature sequence comprises a group of historical regional perception features and the current regional perception feature, and a predicted regional environment feature in the predicted regional perception feature represents a predicted regional environment state in which a target object is expected to be at the next time; selecting a group of target devices matched with the predicted regional environment feature from a group of preset devices, and controlling a device state of each target device to be matched with the predicted regional environment feature.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of smart home, in particular to a device state control method and device, a storage medium and an electronic device. BACKGROUND

[0002] With the wide application of artificial intelligence and Internet of Things technology in the smart home scene, more and more smart home appliances are integrated into people's daily life, becoming a good helper for people's life. The service mode of traditional smart home appliances is mainly voice interaction, which can receive user voice in real time, and through voice recognition and semantic analysis, identify user intent and issue corresponding device operation instructions.

[0003] However, the above-mentioned device state control mode can only act according to the user's instructions, and the service generated according to the user's instructions is passive. In addition, the information obtained by the single-modal interaction mode of voice interaction is one-sided, and cannot form a whole perception of the user and the home environment, so it is difficult to predict the user's real-time needs and respond in time, resulting in low intelligence of the device, and further affecting the user's experience.

[0004] Therefore, the device state control mode in the related art has the problem of low intelligence of the device due to passive control of the device state according to the user's instructions. SUMMARY

[0005] The embodiments of the present application provide a device state control method and device, a storage medium and an electronic device to at least solve the problem of low intelligence of the device due to passive control of the device state according to the user's instructions in the related art.

[0006] According to an aspect of the embodiments of the present application, a device state control method is provided, comprising: obtaining perception data collected by a group of perception sensors in a target region at a current time, to obtain current region perception data, wherein the target region is a region where a target object is located; parsing a current object behavior feature of the target object and a current region environment feature in the target region from the current region perception data, to obtain a region perception feature at the current time, wherein the region perception feature is obtained by fusing the object behavior feature and the region environment feature at the same time; predicting the region perception feature at a next time of the current time according to a region perception feature sequence, to obtain a predicted region perception feature, wherein the region perception feature sequence comprises the region perception feature at a group of historical times and the region perception feature at the current time, and a predicted region environment feature in the predicted region perception feature is used to represent a predicted region environment state in which the target object is expected to be at the next time; selecting a group of target devices matched with the predicted region environment feature from a group of preset devices, and controlling each target device in the group of target devices to be in a device state matched with the predicted region environment feature.

[0007] According to another aspect of the embodiments of the present application, a device state control apparatus is also provided, comprising: an obtaining unit configured to obtain perception data collected by a group of perception sensors in a target region at a current time, to obtain current region perception data, wherein the target region is a region where a target object is located; a parsing unit configured to parse a current object behavior feature of the target object and a current region environment feature in the target region from the current region perception data, to obtain a region perception feature at the current time, wherein the region perception feature is obtained by fusing the object behavior feature and the region environment feature at the same time; a predicting unit configured to predict the region perception feature at a next time of the current time according to a region perception feature sequence, to obtain a predicted region perception feature, wherein the region perception feature sequence comprises the region perception feature at a group of historical times and the region perception feature at the current time, and a predicted region environment feature in the predicted region perception feature is used to represent a predicted region environment state in which the target object is expected to be at the next time; and an executing unit configured to select a group of target devices matched with the predicted region environment feature from a group of preset devices, and control each target device in the group of target devices to be in a device state matched with the predicted region environment feature.

[0008] According to still another aspect of the embodiments of the present application, a computer readable storage medium is also provided, and the computer readable storage medium stores a computer program, wherein the computer program is configured to execute the device state control method when running.

[0009] According to a further aspect of the embodiments of the present application, an electronic device is also provided, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the device state control method by the computer program.

[0010] In the embodiments of the present application, the way of constructing the region perception feature based on the perception data in the current region and then predicting the region perception feature at the next moment is adopted, the perception data collected by a group of perception sensors in the target region at the current moment is obtained to obtain the current region perception data, wherein the target region is the region where the target object is located; the current object behavior feature of the target object and the current region environment feature in the target region are parsed from the current region perception data to obtain the region perception feature at the current moment, wherein the region perception feature is obtained by fusing the object behavior feature and the region environment feature at the same moment; the region perception feature at the next moment of the current moment is predicted according to the region perception feature sequence to obtain the predicted region perception feature, wherein the region perception feature sequence comprises the region perception features at a group of historical moments and the region perception feature at the current moment, and the predicted region environment feature in the predicted region perception feature is used to represent the predicted region environment state in which the target object is expected to be at the next moment; a group of target devices matched with the predicted region environment feature is selected from a group of preset devices, and each target device in the group of target devices is controlled to be in the device state matched with the predicted region environment feature. Since the region perception feature at the next moment is predicted according to the region perception feature at the current moment, and then the target device to be controlled is determined, the purpose of not depending on the user instruction is achieved, the effect of improving the device intelligence is achieved, and the problem of low device intelligence caused by the passive control of the device state according to the user instruction in the device state control mode in the related art is solved. BRIEF DESCRIPTION OF DRAWINGS

[0011] The accompanying drawings, which are incorporated herein and form a part of the specification, illustrate embodiments consistent with the present application and, together with the description, further serve to explain the principles of the application.

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, for those skilled in the art, other drawings can also be obtained based on these drawings without any creative effort.

[0013] Figure 1 is a hardware environment schematic diagram of a device state control method according to an embodiment of the present application;

[0014] Figure 2is a flow diagram of an optional device state control method according to an embodiment of the application;

[0015] Figure 3 is a flow diagram of another optional device state control method according to an embodiment of the application;

[0016] Figure 4 is a flow diagram of yet another optional device state control method according to an embodiment of the application;

[0017] Figure 5 is a flow diagram of an optional device state control method according to an embodiment of the application;

[0018] Figure 6 is a flow diagram of yet another optional device state control method according to an embodiment of the application;

[0019] Figure 7 is a flow diagram of yet another optional device state control method according to an embodiment of the application;

[0020] Figure 8 is a flow diagram of yet another optional device state control method according to an embodiment of the application;

[0021] Figure 9 is a flow diagram of another optional device state control method according to an embodiment of the application;

[0022] Figure 10 is a flow diagram of yet another optional device state control method according to an embodiment of the application;

[0023] Figure 11 is a flow diagram of yet another optional device state control method according to an embodiment of the application;

[0024] Figure 12 is a structural block diagram of an optional device state control apparatus according to an embodiment of the application;

[0025] Figure 13 is a structural block diagram of an optional electronic device according to an embodiment of the application. DETAILED DESCRIPTION

[0026] In order to enable persons skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by persons skilled in the art without creative work should fall within the protection scope of the present application.

[0027] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0028] According to one aspect of the embodiments of this application, a device state control method is provided. This device state control method is widely applicable to whole-house intelligent digital control application scenarios such as smart homes, smart home ecosystems, and intelligence house ecosystems. Optionally, in this embodiment, the above-mentioned device state control method can be applied to, for example... Figure 1 The hardware environment shown consists of terminal device 102 and server 104. For example... Figure 1 As shown, server 104 is connected to terminal device 102 via a network and can be used to provide services (such as application services) to the terminal or clients installed on the terminal. A database can be set up on the server or independently of the server to provide data storage services for server 104. Cloud computing and / or edge computing services can be configured on the server or independently of the server to provide data processing services for server 104.

[0029] The aforementioned network may include, but is not limited to, at least one of the following: wired network, wireless network. The aforementioned wired network may include, but is not limited to, at least one of the following: wide area network, metropolitan area network, local area network. The aforementioned wireless network may include, but is not limited to, at least one of the following: Wi-Fi (Wireless Fidelity), Bluetooth. The terminal device 102 may not be limited to PC, mobile phone, tablet computer, smart air conditioner, smart range hood, smart refrigerator, smart oven, smart stove, smart washing machine, smart water heater, smart washing equipment, smart dishwasher, smart projector, smart TV, smart clothes rack, smart curtains, smart audio-visual equipment, smart socket, smart speaker, smart speaker box, smart fresh air equipment, smart kitchen and bathroom equipment, smart bathroom equipment, smart robot vacuum cleaner, smart window cleaning robot, smart mopping robot, smart air purifier, smart steam oven, smart microwave oven, smart water heater, smart air purifier, smart water dispenser, smart door lock, etc.

[0030] The device state control method of the embodiments of the present application can be executed by the server 104, or jointly executed by the server 104 and the terminal device 102. Taking the server 104 as an example, Figure 2 is a flowchart of an optional device state control method according to an embodiment of the present application, as shown in the figure, the flow of the method can include the following steps: Figure 2

[0031] In step S202, the perception data collected by a group of perception sensors in a target region at a current time is obtained, and current region perception data is obtained, wherein the target region is a region where a target object is located.

[0032] The device state control method in the embodiments can be applied to the scenario of active service of the device. The device here can be the same as the terminal device described above, which is a smart home appliance. The active service can mean a service mode in which the device actively serves the target object by predicting the demand of the target object. The target object can be a user of the device (i.e., a user), or an object in the region where the device is located, which is not limited in the embodiments.

[0033] The service mode of the traditional smart home appliance is mainly voice interaction, which can receive user voice instructions in real time, recognize user intent through voice recognition and semantic analysis, and issue device operation instructions. However, this service mode is passive and not smart enough, and can only act according to user instructions, and cannot actively serve, be more intelligent, and be more considerate by predicting user demand. Specifically, the passive service has the following defects:

[0034] Passive service must be initiated by the user, and if the user does not give voice instructions, the user demand cannot be automatically predicted, the active perception and reminder of the user demand are lacking, the user cannot be cared for like a real person, and the user's growing expectations for the use experience of the smart home appliance cannot be met;

[0035] The information obtained by the single-modal interaction mode of voice interaction is one-sided, and cannot form a whole perception of the user and the home environment, so it is difficult to predict the real-time demand of the user and respond in time;

[0036] The accuracy of the voice model depends on the number and distribution of the training corpus, and the existing corpus structure and words are relatively single and the distribution is relatively concentrated, which cannot meet the generalization needs of the model. At the same time, due to the innovative and diverse characteristics of human language, it is difficult to build new corpus and update the model in time, resulting in poor user experience.

[0037] ​The related technology for predicting user demand currently mainly focuses on the field of online platform services of the Internet, extracts features such as click volume, page dwell time, frequency from user operation logs, constructs a nonlinear model of user behavior and purchase or click commodity, predicts user demand, and thus performs accurate advertisement placement and commodity inventory prediction.

[0038] It is considered that the smart home interaction scene and the online platform interaction scene of the Internet have the following differences:

[0039] The smart home is a 3D space ecological scene, and the user interaction mode is not limited to text, but also has voice, image, video and other modalities of interaction and a large amount of unstructured interaction data, while the user interaction when using the Internet service is mainly text interaction and structured data;

[0040] The user demand of the Internet platform is relatively single (such as purchasing commodities), while the user demand in the smart home scene is more complex, the demand dimension is wider and is strongly related to the home environment state, such as good environmental demand, healthy dietary demand, comfortable sleep demand, quiet reading demand and the like, and each category can be further divided into many more detailed demands, such as good environmental demand including fine-grained demand in multiple dimensions such as volume, light, temperature, air quality and air smell;

[0041] The user demand in the smart home scene is difficult to represent and quantify with structured data, such as the user's demand for a sanitary environment, which is difficult to reuse existing models;

[0042] There is a nonlinear correlation between various user demands in the smart home scene, and it is difficult to decouple individual demands.

[0043] Therefore, there is great limitation in applying the existing technology in the above field of online platform services of the Internet to the smart home field, including but not limited to: the smart home scene and the Internet platform scene have different data distributions, and it is difficult to directly reuse the model; the existing method of extracting features based on structured data to construct a user demand prediction model is difficult to reuse in a large amount of unstructured data scene; the user demand space of the smart home scene is much more complex than that of the online platform of the Internet, and it is difficult to quantify and decouple, so the existing user demand prediction technology cannot be reused.

[0044] To at least partially solve the above-mentioned problems, it is considered that in the smart home scene, the user's needs are met by controlling smart home devices and changing the home environment, so the change of the device state is the embodiment of the user's needs. In the embodiment, unstructured information in multiple modal channels such as images, voices, and videos in the smart home scene can be combined to construct a high-dimensional whole-house perception feature space based on user portraits, environment states, and device states. By predicting the device state at the next time point, the user's needs can be reflected and the home environment can be adjusted in time to discover and solve problems before the user, thereby achieving the purpose of perceiving the user's needs and providing proactive services without the user issuing instructions.

[0045] In the embodiment, before controlling the device state, a set of perception sensors in the target region can first acquire perception data collected at the current time to obtain current regional perception data. Here, the target region is the region where the target object is located, and can be the whole-house region where the target object is located. The set of perception sensors can include multiple different perception sensors.

[0046] In step S204, the current object behavior feature of the target object and the current regional environment feature in the target region are parsed from the current regional perception data to obtain the regional perception feature at the current time, wherein the regional perception feature is obtained by fusing the object behavior feature and the regional environment feature at the same time.

[0047] For the obtained current regional perception data, the current regional perception data can be parsed to parse the current object behavior feature of the target object and the current regional environment feature in the target region from the current regional perception data to obtain the regional perception feature at the current time. Here, the regional perception feature can be obtained by fusing the object behavior feature and the regional environment feature at the same time.

[0048] Optionally, the current object behavior feature of the target object can be determined according to the data feature corresponding to the target object, and can include but is not limited to the current action, emotion, and body state of the target object. The current regional environment feature in the target region can be determined according to the data feature collected in the target region.

[0049] Optionally, the regional perception feature at the current time obtained by fusing the current object behavior feature and the current regional environment feature can be obtained by directly splicing the current object behavior feature and the current regional environment feature.

[0050] For example, taking the regional perception feature as an example, a vectorized expression can be used to merge to form a high-dimensional whole-house perception feature space that can represent real-time home ecological features, i.e., the user information at time t is The whole-house information at time t is The merged high-dimensional whole-house perception feature space at time t is

[0051] At step S206, the next time's regional perception feature at the current time is predicted according to a regional perception feature sequence, to obtain a predicted regional perception feature, wherein the regional perception feature sequence comprises a group of historical time's regional perception features and the current time's regional perception feature, and a predicted regional environment feature in the predicted regional perception feature is used to represent a predicted regional environment state in which the target object is expected to be at the next time;

[0052] In order to improve the accuracy of the prediction of the target object's demand for the regional environment state, a regional perception feature sequence can be constructed by combining the current time's regional perception feature and a group of historical time's regional perception features, and then the next time's regional perception feature at the current time is predicted according to the regional perception feature sequence, to obtain a predicted regional perception feature. Here, the regional perception feature sequence can comprise a group of historical time's regional perception features and the current time's regional perception feature. A predicted regional environment feature in the predicted regional perception feature is used to represent a predicted regional environment state in which the target object is expected to be at the next time.

[0053] For example, taking the current object behavior feature of the target object as user information, and taking the current regional environment feature in the target region as user home environment, the user information, the user home environment, the user historical behavior, and the current event can be combined for information integration and feature extraction, and then the user's future demand can be predicted.

[0054] At step S208, a group of target devices matching the predicted regional environment feature is selected from a group of preset devices, and each target device in the group of target devices is controlled to be in a device state matching the predicted regional environment feature.

[0055] According to the predicted regional environment state in which the target object is expected to be at the next time, a device that needs to be adjusted to achieve the regional environment state can be determined, that is, a group of target devices matching the predicted regional environment feature. In this embodiment, a group of target devices matching the predicted regional environment feature can be selected from a group of preset devices, and each target device in the group of target devices is controlled to be in a device state matching the predicted regional environment feature. Here, the group of preset devices can be a group of devices in the target region.

[0056] For example, in the case of predicting that the user is very hot and needs to reduce the room temperature, the target device can be selected as an air conditioner, and the air conditioner temperature can be adjusted to the user's habitable temperature.

[0057] By the steps S202 to S208, the perception data collected by the group of perception sensors in the target region at the current time is obtained to obtain the current region perception data, wherein the target region is a region where the target object is located; the current object behavior feature of the target object and the current region environment feature in the target region are parsed from the current region perception data to obtain the region perception feature at the current time, wherein the region perception feature is obtained by fusing the object behavior feature and the region environment feature at the same time; the region perception feature at the next time of the current time is predicted according to the region perception feature sequence to obtain the predicted region perception feature, wherein the region perception feature sequence includes the region perception features at a group of historical times and the region perception feature at the current time, and the predicted region environment feature in the predicted region perception feature is used to represent the predicted region environment state in which the target object is expected to be at the next time; a group of target devices matched with the predicted region environment feature is selected from a group of preset devices, and each target device in the group of target devices is controlled to be in a device state matched with the predicted region environment feature, thereby solving the problem of low intelligence of the device caused by passive control of the device state according to the user instruction in the related art.

[0058] In one example embodiment, the region perception feature at the next time of the current time is predicted according to the region perception feature sequence to obtain the predicted region perception feature, including:

[0059] S11, inputting the region perception feature at the current time into a region perception feature prediction model to obtain the predicted region perception feature output by the region perception feature prediction model, wherein the region perception feature prediction model is obtained by model training of an initial region perception feature prediction model using a group of region perception features at a group of historical times and a sample label corresponding to each region perception feature at each historical time in the group of historical times, and the sample label corresponding to each region perception feature at each historical time is used to represent the device state of a group of preset devices at the next time of each historical time.

[0060] In order to improve the prediction accuracy of the predicted region perception feature, the initial region perception feature prediction model can be model trained using a group of region perception features at a group of historical times and a sample label corresponding to each region perception feature at each historical time in the group of historical times to obtain the region perception feature prediction model. After the region perception feature at the current time is determined, the region perception feature at the current time can be input into the region perception feature prediction model to obtain the predicted region perception feature output by the region perception feature prediction model. Here, the sample label corresponding to each region perception feature at each historical time can be used to represent the device state of a group of preset devices at the next time of each historical time.

[0061] For example, taking the area-aware feature prediction model as the time series prediction model, since the state changes of people, things and objects in the home environment extend from the past to the present, the home ecological features at a future time can be predicted by the time series method, that is, the historical data of each sub-module of the aforementioned cognitive reasoning layer is collected to construct a time series prediction model, which can be used to predict the whole-house awareness feature space at a future time. As shown in Figure 3 , the high-dimensional whole-house awareness feature space {F t1 ,F t2, …,F t(n-1)} is predicted to the high-dimensional whole-house awareness feature space {F tn}.

[0062] Through the embodiment, the area-aware feature prediction model is trained in combination with historical area-aware features to predict the area-aware features at the current time, which can improve the accuracy of the prediction results.

[0063] In one example embodiment, a set of target devices matching the predicted area environment features is selected from a set of preset devices, including:

[0064] S21, predicting the device state of each preset device in the set of preset devices according to the predicted area environment features, to obtain the predicted device state of each preset device;

[0065] S22, selecting a set of target devices from the set of preset devices according to the predicted device state of each preset device and the current device state of each preset device, wherein the predicted device state of each target device is different from the current device state of each target device.

[0066] After determining the predicted area environment features, the device state of each preset device in the set of preset devices can be predicted according to the predicted area environment features to obtain the predicted device state of each preset device. When each preset device is in the corresponding predicted device state, the area environment features can be the same as the predicted area environment features.

[0067] For the preset device state of each preset device, the preset device whose preset device state is different from the current device state of each preset device can be determined as a target device, and a set of target devices can be selected from the set of preset devices. Here, the predicted device state of each target device is different from the current device state of each target device. The current device state of each preset device can be reported by each preset device in real time, or can be actively collected by the server, and the embodiment does not limit this.

[0068] By determining the predicted device state of the preset device according to the preset region environment feature, and then determining the target device that needs to be controlled or adjusted, the accuracy of device state adjustment can be improved, and the intelligence of the device can be improved.

[0069] In one example embodiment, the device state of each preset device in a group of preset devices is predicted according to a predicted region environment feature, to obtain a predicted device state of each preset device, including:

[0070] S31, input the predicted region environment feature into the device state prediction model to obtain the predicted device state of each preset device output by the device state prediction model, wherein the device state prediction model is obtained by using a group of region environment feature samples and a sample label corresponding to each region environment feature sample in the group of region environment feature samples to model train an initial device state prediction model, and the sample label corresponding to each region environment feature sample is used to indicate the device state of each preset device corresponding to each region environment feature sample.

[0071] In order to improve the accuracy and prediction efficiency of the prediction result, a device state prediction model can be constructed for predicting the device state. The device state prediction model can be obtained by using a group of region environment feature samples and a sample label corresponding to each region environment feature sample in the group of region environment feature samples to model train an initial device state prediction model. The sample label corresponding to each region environment feature sample can be used to indicate the device state of each preset device corresponding to each region environment feature sample.

[0072] Optionally, the above-mentioned device state prediction model can be completed by using a deep neural network to perform time series modeling on the demand of the target object. In the training stage, a group of region environment feature samples can be input into the device state prediction model, and the sample label corresponding to each region environment feature sample can be output by the device state prediction model. In the prediction stage, the input data of the device state prediction model can be the predicted region environment feature, and the corresponding output data can be the predicted device state of each preset device corresponding to the predicted region environment feature.

[0073] In this embodiment, by inputting the predicted region environment feature into the device state prediction model, the predicted device state of each preset device output by the device state prediction model can be obtained.

[0074] For example, taking the device state prediction model as an example, a deep neural network is used to model the time series of user demand. The input of the model in the training stage is the high-dimensional whole-house perception feature space at the historical time, and the output of the model is the corresponding device state (such as the on-off state, the specific numerical value of the setting, etc.) at each historical time. The use process of the model in the prediction stage can be as shown in Figure 4 The input of the model is the high-dimensional whole-house perception feature space at the next time inferred by the user demand prediction model at the cognitive layer, and the output is the device state parameters at the next time.

[0075] Through this embodiment, by using the device prediction model trained by the regional environment feature sample and the sample label, the predicted device state of each preset device is predicted, which can improve the efficiency of obtaining the predicted device state.

[0076] In one example embodiment, a set of target devices is selected from a set of preset devices according to the predicted device state of each preset device and the current device state of each preset device, including:

[0077] S41, a set of candidate devices is selected from a set of preset devices by comparing the predicted device state of each preset device and the current device state of each preset device, wherein the set of candidate devices is the preset device in the set of preset devices whose predicted device state and current device state are different;

[0078] S42, a set of target events is determined from a set of preset events that have occurred, wherein the set of preset events is an event that is preset and has a correlation relationship with the device state of at least one preset device in the set of preset devices;

[0079] S43, the predicted device state of each candidate device in the set of candidate devices is evaluated for executability according to the set of target events, to obtain an evaluation result corresponding to each candidate device, wherein the evaluation result corresponding to each candidate device is used to indicate whether to allow each candidate device to execute a device state adjustment operation that adjusts the device state to the corresponding predicted device state;

[0080] S44, the candidate device corresponding to the evaluation result indicating that the device state adjustment operation is allowed to be executed is determined as a target device from the set of candidate devices, to obtain a set of target devices

[0081] In order to improve the safety of the device state, and avoid the existing damage to the health of the target object due to the prediction of the device state, in the embodiment, a set of preset events can be set in advance, and the executability of the selected prediction device is evaluated according to the preset event. Here, a set of preset events are events that are set in advance and have a correlation with the device state of at least one preset device in a set of preset devices. The preset event can include events that may conflict with the health of the target object (such as, in the case of a cold, the air conditioner should not be turned on), events that are mutually exclusive with the current device state (such as, in the smart mode, the temperature cannot be adjusted), and events that are absolutely not allowed to occur, which seriously endanger the health of the user and may cause personal danger (such as, in the case of a smoke alarm, the fire is turned on).

[0082] By comparing the prediction device state of each preset device and the current device state of each preset device, a set of candidate devices can be selected from a set of preset devices. At the same time, a set of target events that have occurred in a set of preset events are determined, and the executability of the prediction device state of each candidate device in a set of candidate devices is evaluated according to a set of target events, to obtain an evaluation result corresponding to each candidate device. Here, a set of candidate devices can be preset devices in a set of preset devices whose prediction device state and current device state are different. The evaluation result corresponding to each candidate device can be used to indicate whether each candidate device is allowed to execute the device state adjustment operation of adjusting the device state to the corresponding prediction device state.

[0083] For example, in the case of a smoke alarm, if there is a prediction device state of opening fire, the device state of the prediction device state of opening fire can be evaluated as not allowing the corresponding device to execute the opening fire operation.

[0084] After determining the evaluation results of a set of candidate devices, the candidate devices in a set of candidate devices, whose corresponding evaluation results indicate that the device state adjustment operation is allowed to be executed, can be determined as target devices to obtain a set of target devices.

[0085] Through the embodiment, the executability of the prediction device state is evaluated by the preset event, which can improve the safety of the device state control.

[0086] In one example embodiment, the executability of the prediction device state of each candidate device in a set of candidate devices is evaluated according to a set of target events, to obtain an evaluation result corresponding to each candidate device, including:

[0087] S51, each candidate device is taken as a current candidate device to execute the following executability evaluation operation to obtain an evaluation result corresponding to each candidate device:

[0088] In a case where the predicted device state of the current candidate device has a correlation with a first type of event in the set of target events, the first evaluation result is determined as the evaluation result corresponding to the current candidate device, wherein the first type of event is an event for which a device state having a correlation is not allowed to occur, and the first evaluation result is used to indicate that the device state adjustment operation is not allowed to be performed;

[0089] In a case where the predicted device state of the current candidate device has a correlation with a second type of event in the set of target events, a preset value corresponding to the second type of event is deducted from the initial evaluation value to obtain a target evaluation value, wherein the second type of event is an event for which a device state having a correlation is inhibited from occurring, and the preset value is used to represent the intensity with which the second type of event inhibits the predicted device state of the current candidate device from occurring; in a case where the target evaluation value is greater than or equal to a preset evaluation threshold, the second evaluation result is determined as the evaluation result corresponding to the current candidate device, wherein the second evaluation result is used to indicate that the device state adjustment operation is allowed to be performed; in a case where the target evaluation value is less than the preset evaluation threshold, the third evaluation result is determined as the evaluation result corresponding to the current candidate device, wherein the third evaluation result is used to indicate that the device state adjustment operation is not allowed to be performed;

[0090] In a case where the predicted device state of the current candidate device has no correlation with any target event in the set of target events, the fourth evaluation result is determined as the evaluation result corresponding to the current candidate device, wherein the fourth evaluation result is used to indicate that the device state adjustment operation is allowed to be performed.

[0091] Considering that the risk levels of different preset events are different, the set of preset events can be divided into two types of preset events, the first type of preset event can be an event for which a device state having a correlation is not allowed to occur, and the second type of preset event can be an event for which a device state having a correlation is inhibited from occurring. The evaluation results corresponding to the first type of preset event and the second type of preset event can be different. Correspondingly, the evaluation result of a candidate device can be represented by an evaluation value, and each candidate device can have the same initial evaluation value, i.e., the score of a corresponding device is determined by scoring each device state, and then it is determined whether the device state adjustment operation is allowed to be performed. The lower the score, the less likely it is that the device state adjustment operation is allowed to be performed.

[0092] In the embodiment, the executability evaluation operation can be performed on each candidate device as a current candidate device to obtain an evaluation result corresponding to each candidate device. The executability evaluation operation can include: in a case where the predicted device state of the current candidate device has an association relationship with a first type of event in the set of target events, determining a first evaluation result as the evaluation result corresponding to the current candidate device. Here, the first evaluation result can be used to indicate that the device state adjustment operation is not allowed to be performed. Correspondingly, when it is determined that the predicted device state of the current candidate device has an association relationship with the first type of event in the set of target events, the initial evaluation value of the current candidate device can be directly adjusted to the minimum value, such as 0.

[0093] Optionally, in a case where the first evaluation result is determined as the evaluation result corresponding to the current candidate device, the first type of event and the corresponding predicted device state can be reminded according to the first evaluation result to avoid the target object actively adjusting the current candidate device to the predicted device state, thereby generating a danger.

[0094] The executability evaluation operation can further include: in a case where the predicted device state of the current candidate device has an association relationship with a second type of event in the set of target events, deducting a preset value corresponding to the second type of event from the initial evaluation value to obtain a target evaluation value. In a case where the target evaluation value is greater than or equal to a preset evaluation threshold, a second evaluation result is determined as the evaluation result corresponding to the current candidate device. Here, the preset value can be used to represent the intensity of the second type of event inhibiting the predicted device state of the current candidate device. The second evaluation result can be used to indicate that the device state adjustment operation is allowed to be performed.

[0095] Optionally, in a case where the second evaluation result is determined as the evaluation result corresponding to the current candidate device, the second type of event and the corresponding predicted device state can be reminded according to the second evaluation result to prompt the target object that the current candidate device will be adjusted to the predicted device state.

[0096] In a case where the target evaluation value is less than the preset evaluation threshold, a third evaluation result can be determined as the evaluation result corresponding to the current candidate device. Here, the third evaluation result is used to indicate that the device state adjustment operation is not allowed to be performed.

[0097] Optionally, in a case where the third evaluation result is determined as the evaluation result corresponding to the current candidate device, the second type of event and the corresponding predicted device state can be reminded according to the third evaluation result to select whether the current candidate device is adjusted to the predicted device state by the target object. It should be noted that the foregoing various reminding manners can include but are not limited to text reminders, voice reminders, etc. For example, Figure 5As shown, after determining the target device and the predicted device state of the target device according to the evaluation result, a related reminder can be given to the target object, such as "smart gas stove now wind three", "no meeting arrangement today, schedule reminder has "dental visit", do you need to be contacted by phone?" "smart gas stove now wind three", "oven is in prediction" and the like.

[0098] In the case where the predicted device state of the current candidate device has no association with any target event in the set of target events, the fourth evaluation result can be determined as the evaluation result corresponding to the current candidate device. Here, the fourth evaluation result is used to indicate that the device state adjustment operation is allowed to be performed.

[0099] For example, taking the user as the target object, after predicting the actual demand of the user based on the current high-dimensional whole-house perception feature, the user's actual demand can be matched and scored with the device services executable in the current home ecological environment, and the device service with the highest score is issued as the solution to solve the user's pain points and meet the user's demand. In the process of predicting user demand and matching device services, the user's health and the limitations of the device state need to be considered. For example, when reasoning the user's demand, the user's demand is to set the air conditioner temperature to 24 degrees, but the user is currently suffering from a cold, and the room temperature should not be too low, so the user needs to be reminded in a timely manner, or when matching the device service, the air conditioner temperature adjustment service has the highest matching score, but the air conditioner is currently in smart mode and does not support temperature adjustment, so it needs to be confirmed with the user whether to close the smart mode.

[0100] Optionally, the execution of the above-mentioned executability evaluation operation can be completed by a device service matching and scoring submodule of the service decision layer. When the submodule executes the above-mentioned executability evaluation operation, it can first judge the association between the predicted device state of the current candidate device and the first type of event, and then judge the association between the predicted device state of the current candidate device and the second type of event in the case where there is no association. According to the number of devices of a set of preset devices and the number of events of the first type of event and the second type of event, a deduction matrix can be constructed respectively, and according to the different deduction mechanisms of the first type of event and the second type of event, the final score of the current candidate preset device can be determined, and then the evaluation result of the current candidate preset device can be determined. It should be noted that the service decision layer can include three submodules, i.e., user demand prediction, device service matching and scoring, and logical decision, as shown in the following figure. Figure 6 As shown, the user demand prediction submodule is used to predict the aforementioned predicted area perception feature, and the logical decision submodule is used to select the target device according to the evaluation result.

[0101] For example, in order to perform the executability evaluation operation, a first-level decision knowledge base and a second-level decision knowledge base can be constructed respectively. The first-level decision knowledge base is used to store the above-mentioned first type of event, and the second-level decision knowledge base is used to store the above-mentioned second type of event. For example,Figure 7 As shown, the current state of each device is scored according to the first and second knowledge bases. The initial score of each device is 100%, and the events in the first knowledge base can be designed as different deduction mechanisms. Once the events in the second knowledge base occur, the score of the device is 0. The final output is the score of each device.

[0102] Suppose there are K devices in the user's home. The number of the first knowledge base M is Q, and the deduction matrix is MQ. Each number in the matrix represents the score to be deducted after the occurrence of the event. The number of the second knowledge base N is P. If the device hits any one of the N P∈{1,…,P} events, the score of the device is 0, i.e. N P∈{1,…,P} = 0 or 1. The final score of the device is calculated According to the output score of the device i, it can be determined whether the score of the device exceeds the threshold value, as shown in Figure 8 . If yes, the device i is controlled to execute the predicted device state parameter, otherwise it is not executed.

[0103] Through this embodiment, by classifying a group of preset events into two types of preset events, and determining the executability of the predicted device state of the current candidate device according to the association relationship between the current candidate device and the two types of preset events, the safety and intelligence of the device state adjustment can be improved.

[0104] In one example embodiment, a set of perception sensors in a target area collect perception data at a current time to obtain current area perception data, including at least one of:

[0105] S61, acquiring image data collected by an image acquisition device in the target area at the current time to obtain current image data;

[0106] S62, acquiring audio data collected by a sound acquisition device in the target area at the current time to obtain current audio data;

[0107] S63, acquiring smell data collected by a smell acquisition device in the target area at the current time to obtain current smell data;

[0108] S64, acquiring pressure data collected by a pressure sensor in the target area at the current time to obtain current pressure data;

[0109] S65, acquiring smoke data collected by a smoke sensor in the target area at the current time to obtain current smoke data;

[0110] S66, acquiring temperature and humidity data collected by a temperature and humidity sensor in the target area at the current time to obtain current temperature and humidity data.

[0111] To improve the richness of the current regional perception data, and thus improve the prediction accuracy of the predicted regional perception features, in the embodiment, the group of perception sensors in the target region can include at least one of the following: image acquisition device (such as camera), sound acquisition device (such as microphone), smell acquisition device (such as electronic nose), pressure sensor, smoke sensor, temperature and humidity sensor. Here, the temperature and humidity sensor can include both temperature and humidity sensors, which can be integrated or separated, and the embodiment does not limit this.

[0112] Correspondingly, obtaining the perception data collected by the group of perception sensors in the target region at the current time to obtain the current regional perception data can refer to at least one of the following: obtaining image data collected by the image acquisition device in the target region at the current time to obtain the current image data; obtaining audio data collected by the sound acquisition device in the target region at the current time to obtain the current audio data; obtaining smell data collected by the smell acquisition device in the target region at the current time to obtain the current smell data; obtaining pressure data collected by the pressure sensor in the target region at the current time to obtain the current pressure data; obtaining smoke data collected by the smoke sensor in the target region at the current time to obtain the current smoke data; obtaining temperature and humidity data collected by the temperature and humidity sensor in the target region at the current time to obtain the current temperature and humidity data.

[0113] Optionally, in the embodiment, a multi-modal information perception layer can be constructed, as shown in the following figure: Figure 9 As shown in the figure, the multi-modal information of the user and the home environment such as image, video stream, voice, smell, pressure, smoke, etc. is received by the camera, microphone, electronic nose, pressure sensor, smoke sensor, temperature sensor, etc. Hardware device, convert the information into data expression symbol that computer can understand, input into multi-modal cognitive inference layer for analysis, generate high-dimensional whole house perception feature space representing home ecological environment state.

[0114] Through the embodiment, the perception data is obtained by a group of sensors, which can improve the richness of the current regional perception data, and thus improve the accuracy of the prediction result.

[0115] In one example embodiment, the current object behavior feature of the target object and the current regional environment feature in the target region are parsed from the current regional perception data to obtain the regional perception feature at the current time, including:

[0116] S71, performing at least one of the following image analysis operations on the current image data in the current regional perception data to obtain the first behavior feature of the target object at the current time: face recognition, human action detection, object recognition, optical character recognition;

[0117] S72, performing a voice analysis operation on the current voice data in the current region perception data to obtain a second behavior feature of the target object at present, the voice analysis operation including at least one of voice recognition, emotion recognition, voiceprint recognition, and specified event recognition.

[0118] S73, performing a text analysis operation on the current text data in the current region perception data to obtain a third behavior feature of the target object at present, the text analysis operation including at least one of semantic understanding, syntax structure analysis, and key entity word extraction.

[0119] S74, extracting a current region environment feature in the target region from the current region perception data.

[0120] For the acquired region perception feature at the current time, corresponding data analysis operations can be respectively performed according to data features of the perception data to obtain the current object behavior feature of the target object. In this embodiment, for the current image data in the current region perception data, an image analysis operation can be performed to obtain the first behavior feature of the target object at present. The image analysis operation can include at least one of face recognition, human body action detection, object recognition, and optical character recognition. The optical character recognition can be OCR (Optical Character Recognition) recognition.

[0121] For the current voice data in the current region perception data, a voice analysis operation can be performed to obtain the second behavior feature of the target object at present. The voice analysis operation can include at least one of voice recognition, emotion recognition, voiceprint recognition, and specified event recognition. The specified event can be an event related to the health state of the target object, which can include but is not limited to cough detection, help detection, etc.

[0122] For the current text data in the current region perception data, a text analysis operation can be performed to obtain the third behavior feature of the target object at present. The text analysis operation can include at least one of semantic understanding, syntax structure analysis, and key entity word extraction.

[0123] In this embodiment, for the current region environment feature in the target region, the current region environment feature in the target region can be directly extracted from the current region perception data.

[0124] Optionally, the process of determining the region awareness feature at the current time can be completed in the multi-modal cognitive inference layer. The multi-modal cognitive inference layer here can include two modules of user information and whole house information. The user information module (i.e., the module of obtaining the current object behavior feature) performs the most superficial and direct analysis on the user's body movements, expressions, voice semantics, emotional tone, etc. through the CV (Computer Vision) technology, NLP (Natural Language Processing) technology, voice technology, so as to understand the user's current posture, what the user is doing, what the user is saying, but not to analyze the meaning behind the user's behavior. The whole house information module (i.e., the module of obtaining the current region environment feature) combines the user information module, the user's home environment, the user's historical behavior, and the current event to integrate information and extract features, forming real-time perception information of the whole house at the current time, including the current user.

[0125] For example, taking the target object as a user, the user says "I am hungry, should I eat something or not, it is very troublesome at late night", the user information module detects the identity of the user through face recognition, the user is full of sweat, the NLP sub-module analyzes that the user wants to eat, and the voice sub-module analyzes that the user is very hesitant. The whole house information module combines the user portrait corresponding to the user's identity and the whole house perception information to predict the user's demand. For example, it is known from the user portrait that the user has high body fat, and has been reducing fat in recent days. The user mentioned "night" and "trouble" in the above text, and the whole house perception module identifies that the user's home has a cooking tool such as a cooking range and food materials such as vegetables in the refrigerator.

[0126] Through the embodiment, the current object behavior feature of the target object is analyzed through images, voice, text, etc., which can improve the accuracy of analyzing the current behavior of the target object, and further improve the accuracy of the prediction result.

[0127] In one example embodiment, controlling each target device in a group of target devices to be in a device state matching the predicted region environment feature comprises:

[0128] S81, generating a corresponding device control instruction for each target device according to the device state matching the predicted region environment feature of each target device;

[0129] S82, sending the corresponding device control instruction generated for each target device to each target device to control each target device to be in a device state matching the predicted region environment feature.

[0130] In order to ensure the timeliness of the active service of the device, after a set of target devices and the device state corresponding to each target device are determined, a device control instruction corresponding to each target device can be generated according to the device state matched with the predicted regional environmental feature of each target device, and the device control instruction corresponding to each target device is sent to each target device to control each target device to be in the device state matched with the predicted regional environmental feature.

[0131] Through this embodiment, the efficiency of adjusting the device state can be improved by controlling the target device to adjust the device state through the device control instruction.

[0132] The device state control method in this embodiment will be explained in combination with an optional example. In this optional example, the target region is a whole house region, the regional perception feature at the current time is a high-dimensional whole house perception feature, and the target object is a user.

[0133] The optional example provides a user demand prediction method and system based on multi-modal whole house perception. Through the fusion of unstructured information in multiple modal channels such as images, voices, and videos in the house, a high-dimensional whole house perception feature space based on user portraits, environmental states, and device states is constructed. The device state at the next time point is predicted to reflect the user demand and make timely adjustments to the home environment. The problem is solved before the user finds it, thereby creating a high-order intelligent home ecological system that integrates multiple modal interactions, perceives user needs without user instruction, and does not rely on a training corpus.

[0134] As shown in Figure 10 and Figure 11 The device state control method in this optional example can include a multi-modal information perception layer, a multi-modal cognitive reasoning layer, a service decision layer, and a service execution layer. The flow of the device state control method in this optional example can include the following steps:

[0135] Step 1: The multi-modal information perception layer receives video, image, voice, and other multi-modal information of the user and the home environment through the camera, microphone, and various sensors, and converts the information into data expression symbols that can be understood by the computer.

[0136] Step 2: The multi-modal cognitive reasoning layer analyzes the user information and integrates the user information with the user's home environment, user historical behavior, and current events to extract features and obtain surface and hidden layer information, forming real-time perception information of the whole house at the current time, including the current user.

[0137] Step 3, the user demand is obtained by the service decision layer through the information reasoning of the cognitive reasoning layer, and the device service satisfying the user demand is matched, and then whether the device service can be issued is discriminated according to the conflict judgment knowledge base, and the executable device service instruction is generated and issued.

[0138] Step 4, the device instruction output by the service decision layer is issued and executed by the instruction execution layer.

[0139] Through the optional example, a large amount of unstructured multi-modal information is integrated into the user home ecological environment characteristic space, without decoupling the user demand and the environment characteristics, the user demand prediction model is constructed according to the data change trend in the history time, and the device instruction solving the user demand is output, the high-order intelligence that the intelligent household electrical appliances actively perceive and remind the user demand by using the multi-modal information is realized, the training of the model does not depend on the user corpus, and the problem of frequent misrecognition of the user intention due to lack of corpus generalization is avoided, and the interactive experience of the user is optimized.

[0140] It should be noted that, for the foregoing method embodiments, in order to simply describe, they are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited to the action sequence described, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions and modules involved are not necessarily essential to the present application.

[0141] From the above description of the embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be realized by means of software and the necessary general hardware server, and of course, it can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as a ROM (Read-Only Memory), a RAM (Random Access Memory), a magnetic disk, or an optical disk), and includes a plurality of instructions for causing an end device (which can be a mobile phone, a computer, a server, or a network device) to execute the method of each embodiment of the present application.

[0142] According to another aspect of the embodiments of the present application, a device state control apparatus for implementing the above-mentioned device state control method is also provided, which can be applied to a smart device. Figure 12 is a structural block diagram of an optional device state control apparatus according to an embodiment of the present application, as shown in Figure 12 the apparatus can include:

[0143] The acquisition unit 1202 is configured to acquire perception data collected by a group of perception sensors in a target region at a current time, to obtain current regional perception data, wherein the target region is a region where the target object is located.

[0144] The analysis unit 1204 is connected with the acquisition unit 1202 and is configured to analyze the target object's current object behavior feature and the current regional environment feature in the target region from the current regional perception data, to obtain regional perception features at the current time, wherein the regional perception features are obtained by fusing the object behavior feature and the regional environment feature at the same time.

[0145] The prediction unit 1206 is connected with the analysis unit 1204 and is configured to predict the regional perception features at a next time according to a sequence of regional perception features, to obtain predicted regional perception features, wherein the sequence of regional perception features includes the regional perception features at a group of historical times and the regional perception features at the current time, and the predicted regional environment feature in the predicted regional perception features is used to represent a predicted regional environment state in which the target object is expected to be at the next time.

[0146] The execution unit 1208 is connected with the prediction unit 1206 and is configured to select a group of target devices matched with the predicted regional environment feature from a group of preset devices, and control each target device in the group of target devices to be in a device state matched with the predicted regional environment feature.

[0147] It should be noted that the acquisition unit 1202 in this embodiment can be configured to perform the above step S202, the analysis unit 1204 in this embodiment can be configured to perform the above step S204, the prediction unit 1206 in this embodiment can be configured to perform the above step S206, and the execution unit 1208 in this embodiment can be configured to perform the above step S208.

[0148] The above module obtains a group of perception data collected by a group of perception sensors in a target region at a current moment, to obtain current region perception data, wherein the target region is a region where a target object is located; a current object behavior feature of the target object and a current region environment feature in the target region are parsed from the current region perception data, to obtain a region perception feature at the current moment, wherein the region perception feature is obtained by fusing the object behavior feature and the region environment feature at the same moment; a region perception feature at a next moment of the current moment is predicted according to a region perception feature sequence, to obtain a predicted region perception feature, wherein the region perception feature sequence includes the region perception features at a group of historical moments and the region perception feature at the current moment, and a predicted region environment feature in the predicted region perception feature is used to represent a predicted region environment state in which the target object is expected to be at the next moment; a group of target devices that match the predicted region environment feature are selected from a group of preset devices, and each target device in the group of target devices is controlled to be in a device state that matches the predicted region environment feature, thereby solving the problem in the related art that the device state control mode can only passively control the device state according to a user instruction, resulting in low intelligence of the device, and improving the intelligence of the device.

[0149] In one example embodiment, the prediction unit includes:

[0150] The input module is configured to input the region perception feature at the current moment into the region perception feature prediction model, to obtain the predicted region perception feature output by the region perception feature prediction model, wherein the region perception feature prediction model is obtained by using a group of historical region perception features and sample labels corresponding to each historical region perception feature in the group of historical region perception features to model train an initial region perception feature prediction model, and the sample label corresponding to each historical region perception feature is used to represent a device state of the group of preset devices at a next moment of each historical moment.

[0151] In one example embodiment, the execution unit includes:

[0152] The prediction module is configured to predict the device state of each preset device in the group of preset devices according to the predicted region environment feature, to obtain a predicted device state of each preset device.

[0153] The selection module is configured to select a group of target devices from the group of preset devices according to the predicted device state of each preset device and a current device state of each preset device, wherein the predicted device state of each target device is different from the current device state of each target device.

[0154] In one example embodiment, the prediction module includes:

[0155] The input submodule is configured to input the predicted area environment features into the device state prediction model to obtain a predicted device state of each preset device output by the device state prediction model, wherein the device state prediction model is obtained by training an initial device state prediction model using a set of area environment feature samples and sample labels corresponding to each area environment feature sample in the set of area environment feature samples, and each sample label corresponding to each area environment feature sample is used to indicate a device state of each preset device corresponding to each area environment feature sample.

[0156] In an example embodiment, the selection module includes:

[0157] The selection submodule is configured to select a set of candidate devices from the set of preset devices by comparing the predicted device state of each preset device with the current device state of each preset device, wherein the set of candidate devices are preset devices in the set of preset devices whose predicted device state and current device state are different.

[0158] The determination submodule is configured to determine a set of target events that have occurred from a set of preset events, wherein the set of preset events are preset events that have a correlation with the device state of at least one preset device in the set of preset devices.

[0159] The evaluation submodule is configured to perform an executability evaluation on the predicted device state of each candidate device in the set of candidate devices according to the set of target events to obtain an evaluation result corresponding to each candidate device, wherein the evaluation result corresponding to each candidate device is used to indicate whether the device state adjustment operation of adjusting the device state to the corresponding predicted device state is allowed to be performed by each candidate device.

[0160] The determination submodule is configured to determine a candidate device in the set of candidate devices for which the corresponding evaluation result indicates that the device state adjustment operation is allowed to be performed as a target device to obtain a set of target devices.

[0161] In an example embodiment, the evaluation submodule includes:

[0162] The execution subunit is configured to perform the following executability evaluation operation on each candidate device as a current candidate device to obtain the evaluation result corresponding to each candidate device:

[0163] In a case where the predicted device state of the current candidate device has a correlation with a first type of event in the set of target events, a first evaluation result is determined as the evaluation result corresponding to the current candidate device, wherein the first type of event is an event that does not allow the device state having the correlation to appear, and the first evaluation result is used to indicate that the device state adjustment operation is not allowed to be performed.

[0164] In a case where the predicted device state of the current candidate device has a correlation with a second type of event in the set of target events, a preset value corresponding to the second type of event is deducted from the initial evaluation value to obtain a target evaluation value, the second type of event is an event that inhibits the occurrence of the device state having the correlation, and the preset value is used to represent the intensity of the second type of event inhibiting the occurrence of the predicted device state of the current candidate device; in a case where the target evaluation value is greater than or equal to a preset evaluation threshold, the second evaluation result is determined as the evaluation result corresponding to the current candidate device, the second evaluation result is used to indicate that the device state adjustment operation is allowed to be performed; in a case where the target evaluation value is less than the preset evaluation threshold, the third evaluation result is determined as the evaluation result corresponding to the current candidate device, and the third evaluation result is used to indicate that the device state adjustment operation is not allowed to be performed.

[0165] In a case where the predicted device state of the current candidate device has no correlation with any target event in the set of target events, the fourth evaluation result is determined as the evaluation result corresponding to the current candidate device, and the fourth evaluation result is used to indicate that the device state adjustment operation is allowed to be performed.

[0166] In an example embodiment, the obtaining unit includes at least one of the following:

[0167] The first obtaining module is configured to obtain image data collected by an image collection device in the target area at the current time to obtain current image data.

[0168] The second obtaining module is configured to obtain audio data collected by a sound collection device in the target area at the current time to obtain current audio data.

[0169] The third obtaining module is configured to obtain smell data collected by a smell collection device in the target area at the current time to obtain current smell data.

[0170] The fourth obtaining module is configured to obtain pressure data collected by a pressure sensor in the target area at the current time to obtain current pressure data.

[0171] The fifth obtaining module is configured to obtain smoke data collected by a smoke sensor in the target area at the current time to obtain current smoke data.

[0172] The sixth obtaining module is configured to obtain temperature and humidity data collected by a temperature and humidity sensor in the target area at the current time to obtain current temperature and humidity data.

[0173] In an example embodiment, the analysis unit includes:

[0174] The first execution module is used to perform at least one of the following image parsing operations on the current image data in the current area perception data to obtain the current first behavioral feature of the target object: face recognition, human action detection, object recognition, and optical character recognition;

[0175] The second execution module is used to perform at least one of the following speech parsing operations on the current speech data in the current area perception data to obtain the current second behavioral feature of the target object: speech recognition, emotion recognition, voiceprint recognition, and specified event recognition;

[0176] The third execution module is used to perform at least one of the following text parsing operations on the current text data in the current area perception data to obtain the current third behavioral features of the target object: semantic understanding, syntactic structure analysis, and key entity word extraction.

[0177] The extraction module is used to extract the current regional environmental features within the target area from the current area perception data.

[0178] In one exemplary embodiment, the execution unit includes:

[0179] The generation module is used to generate corresponding device control instructions for each target device based on the device status that matches the environmental characteristics of the predicted area with each target device.

[0180] The sending module is used to send the corresponding device control command generated for each target device to each target device, so as to control each target device to be in a device state that matches the environmental characteristics of the predicted area.

[0181] According to another aspect of the embodiments of this application, a storage medium is also provided, which can be located on a smart device. Optionally, in this embodiment, the storage medium can be used to execute program code of any of the device state control methods described above in the embodiments of this application.

[0182] Optionally, in this embodiment, the storage medium may be located on at least one of the network devices in the network shown in the above embodiment.

[0183] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps:

[0184] S1, acquire the perception data collected by a set of perception sensors in the target area at the current moment, and obtain the current area perception data, where the target area is the area where the target object is located;

[0185] S2, parse the current object behavior feature of the target object and the current regional environment feature in the target region from the current regional perception data, to obtain the regional perception feature at the current moment, wherein the regional perception feature is obtained by fusing the object behavior feature and the regional environment feature at the same moment;

[0186] S3, predicting the regional perception feature at the next moment according to the regional perception feature sequence, to obtain the predicted regional perception feature, wherein the regional perception feature sequence comprises a group of regional perception features at historical moments and the regional perception feature at the current moment, and the predicted regional environment feature in the predicted regional perception feature is used to represent the predicted regional environment state in which the target object is expected to be at the next moment;

[0187] S4, selecting a group of target devices matched with the predicted regional environment feature from a group of preset devices, and controlling each target device in the group of target devices to be in a device state matched with the predicted regional environment feature.

[0188] Optionally, specific examples in the embodiment can refer to the examples described in the above embodiments, and details are not described herein.

[0189] Optionally, in the embodiment, the storage medium can include but is not limited to a U disk, a ROM, a RAM, a mobile hard disk, a magnetic disk or an optical disk and various storage program codes.

[0190] According to another aspect of the embodiment of the application, an electronic device for implementing the above device state control method is also provided, and the electronic device can be a smart device, which can be a server, a terminal or a combination thereof.

[0191] Figure 13 is a structural block diagram of an optional electronic device according to the embodiment of the application, as shown in Figure 13 the processor 1302, the communication interface 1304 and the memory 1306 complete communication with each other through the communication bus 1308, wherein,

[0192] the memory 1306 is used for storing a computer program;

[0193] the processor 1302 is used for executing the computer program stored in the memory 1306, and the following steps are implemented:

[0194] S1, obtaining perception data collected by a group of perception sensors in a target region at a current moment, to obtain current regional perception data, wherein the target region is a region where a target object is located;

[0195] S2, a current object behavior feature of the target object and a current regional environment feature in the target region are parsed from the current regional perception data, to obtain a regional perception feature at the current moment, wherein the regional perception feature is obtained by fusing the object behavior feature and the regional environment feature at the same moment;

[0196] S3, a next moment regional perception feature at the current moment is predicted according to a regional perception feature sequence, to obtain a predicted regional perception feature, wherein the regional perception feature sequence comprises a group of historical moment regional perception features and the current moment regional perception feature, and a predicted regional environment feature in the predicted regional perception feature is used to represent a predicted regional environment state in which the target object is expected to be at the next moment;

[0197] S4, a group of target devices matched with the predicted regional environment feature are selected from a group of preset devices, and each target device in the group of target devices is controlled to be in a device state matched with the predicted regional environment feature.

[0198] Optionally, the communication bus can be a PCI (Peripheral Component Interconnect) bus, an EISA (Extended Industry Standard Architecture) bus, or the like. The communication bus can be divided into an address bus, a data bus, a control bus, and the like. For ease of representation, Figure 13 Only one thick line is used in the figure, but it does not mean that there is only one bus or only one type of bus. The communication interface is used for communication between the electronic device and other devices.

[0199] The memory can include a RAM and can also include a non-volatile memory, for example, at least one disk memory. Optionally, the memory can also be at least one storage device located away from the aforementioned processor.

[0200] As an example, the aforementioned memory 1306 can include but is not limited to the acquisition unit 1202, the parsing unit 1204, the prediction unit 1206, and the execution unit 1208 in the device state control apparatus. In addition, other module units in the device state control apparatus can also be included, but not limited to, which will not be described in detail in this example.

[0201] The processors mentioned above can be general-purpose processors, including but not limited to: CPU (Central Processing Unit), NP (Network Processor), etc.; they can also be DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0202] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments, and will not be repeated here.

[0203] Those skilled in the art will understand that Figure 13 The structure shown is for illustrative purposes only. The device implementing the above device state control method can be a terminal device, such as a smartphone (e.g., Android phone, iOS phone), tablet computer, PDA, mobile Internet Devices (MID), PAD, etc. Figure 13 This does not limit the structure of the aforementioned electronic device. For example, the electronic device may also include components that are more... Figure 13 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 13 The different configurations shown.

[0204] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, ROM, RAM, disk or optical disk, etc.

[0205] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0206] The integrated units in the above embodiments, if implemented in the form of software function units and sold or used as independent products, can be stored in the above computer-readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the parts that make contributions to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for causing one or more computer devices (which can be personal computers, servers or network devices, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application.

[0207] In the above embodiments of the present application, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0208] In the several embodiments provided in the present application, it should be understood that the disclosed client can be implemented by other means. Among them, the above-described device embodiments are only schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the shown or discussed mutual units can be indirect coupling or communication connection through some interface, unit or module, and can be electrical or other forms.

[0209] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place or distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the scheme provided in the embodiments.

[0210] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or at least two units can be integrated in one unit. The above integrated unit can be realized in the form of hardware or in the form of software function unit.

[0211] The above is only the preferred embodiment of the present application, and it should be pointed out that for those skilled in the art, without departing from the principle of the present application, a number of improvements and refinements can be made, which should be regarded as the protection scope of the present application.

Claims

1. A method for controlling the state of equipment, characterized in that, include: The sensor data collected by a set of sensors in the target area at the current moment is obtained, and the predicted area perception features at the next moment are predicted. The predicted area environment features in the predicted area perception features are used to represent the predicted area environment state that the target object is expected to be in at the next moment. Based on the predicted area environmental characteristics, a set of candidate devices is selected, and each candidate device in the set is used as the current candidate device to perform the following executability evaluation operation to obtain the evaluation result corresponding to each candidate device: If the predicted device state of the current candidate device is correlated with a first type of event in a set of target events that have already occurred, the first evaluation result is determined as the evaluation result corresponding to the current candidate device, wherein the first type of event is an event that does not allow correlated device states to occur, and the first evaluation result is used to indicate that the device state adjustment operation is not allowed; If the predicted device state of the current candidate device is correlated with a second type of event in the set of target events, a preset value corresponding to the second type of event is subtracted from the initial evaluation value to obtain the target evaluation value, wherein the second type of event is an event that suppresses correlated device states. The event of the standby state occurs, and the preset value is used to represent the strength of the second type of event in suppressing the predicted device state of the current candidate device; when the target evaluation value is greater than or equal to the preset evaluation threshold, the second evaluation result is determined as the evaluation result corresponding to the current candidate device, wherein the second evaluation result is used to indicate that the device state adjustment operation is allowed; when the target evaluation value is less than the preset evaluation threshold, the third evaluation result is determined as the evaluation result corresponding to the current candidate device, wherein the third evaluation result is used to indicate that the device state adjustment operation is not allowed; when the predicted device state of the current candidate device is not associated with any of the target events in the set of target events, the fourth evaluation result is determined as the evaluation result corresponding to the current candidate device, wherein the fourth evaluation result is used to indicate that the device state adjustment operation is allowed. The evaluation results of the candidate devices in the set of candidate devices are used to indicate which candidate devices are allowed to perform device state adjustment operations and determine them as target devices, thereby obtaining a set of target devices, and controlling each target device in the set of target devices to be in a device state that matches the environmental characteristics of the predicted area.

2. The method according to claim 1, characterized in that, The predicted region-aware features for the next time step are obtained from the current time step, including: The current object behavior characteristics of the target object and the current regional environment characteristics within the target area are parsed from the collected current area perception data to obtain the area perception characteristics at the current moment. The area perception characteristics are obtained by fusing the object behavior characteristics and regional environment characteristics at the same moment. The target area is the area where the target object is located. Based on the region-aware feature sequence, the region-aware features for the next time step are predicted to obtain the predicted region-aware features. The region-aware feature sequence includes a set of region-aware features from historical time steps and the region-aware features for the current time step.

3. The method according to claim 2, characterized in that, The step of predicting the region-aware features for the next time step based on the region-aware feature sequence to obtain predicted region-aware features includes: The current area perception feature is input into the area perception feature prediction model to obtain the predicted area perception feature output by the area perception feature prediction model. The area perception feature prediction model is obtained by training an initial area perception feature prediction model using the area perception features of the set of historical times and the sample labels corresponding to the area perception features of each historical time in the set of historical times. The sample labels corresponding to the area perception features of each historical time are used to represent the device state of a set of preset devices at the next time of each historical time.

4. The method according to claim 1, characterized in that, A set of candidate devices is selected based on the predicted regional environmental characteristics, including: Based on the environmental characteristics of the predicted area, the device status of each preset device in a set of preset devices is predicted to obtain the predicted device status of each preset device. By comparing the predicted device state of each preset device with the current device state of each preset device, a set of candidate devices is selected from the set of preset devices, wherein the set of candidate devices are preset devices whose predicted device state and current device state are different from those of the set of preset devices.

5. The method according to claim 4, characterized in that, The step of predicting the device status of each preset device in a set of preset devices based on the environmental characteristics of the predicted area, to obtain the predicted device status of each preset device, includes: The predicted regional environmental features are input into the device status prediction model to obtain the predicted device status of each preset device output by the device status prediction model. The device status prediction model is obtained by training the initial device status prediction model using a set of regional environmental feature samples and sample labels corresponding to each regional environmental feature sample in the set of regional environmental feature samples. The sample labels corresponding to each regional environmental feature sample are used to indicate the device status of each preset device corresponding to each regional environmental feature sample.

6. The method according to claim 1, characterized in that, The method further includes: A set of target events that have occurred in a set of preset events is determined. The set of preset events are pre-set events that are associated with the device state of at least one preset device in the set of preset devices. The evaluation result corresponding to each candidate device is used to indicate whether each candidate device is allowed to perform a device state adjustment operation to adjust the device state to the corresponding predicted device state. The set of candidate devices are the devices in the set of preset devices.

7. The method according to claim 1, characterized in that, The acquisition of sensing data collected by a set of sensing sensors within the target area at the current moment includes at least one of the following: Obtain the image data acquired by the image acquisition device within the target area at the current time to obtain the current image data; Obtain the audio data collected by the sound acquisition device within the target area at the current moment to obtain the current audio data; Obtain the odor data collected by the odor collection device in the target area at the current time to obtain the current odor data; Obtain the pressure data collected by the pressure sensor within the target area at the current moment to obtain the current pressure data; Obtain the smoke data collected by the smoke sensors in the target area at the current moment to obtain the current smoke data; The temperature and humidity data collected by the temperature and humidity sensors in the target area at the current time are obtained to obtain the current temperature and humidity data.

8. The method according to claim 2, characterized in that, The process of parsing the current object behavior features of the target object and the current regional environment features within the target area from the collected current regional perception data to obtain the regional perception features at the current moment includes: Perform at least one of the following image parsing operations on the current image data in the current area perception data to obtain the first behavioral feature of the target object: face recognition, human motion detection, object recognition, and optical character recognition; Perform at least one of the following speech parsing operations on the current speech data in the current area perception data to obtain the current second behavioral feature of the target object: speech recognition, emotion recognition, voiceprint recognition, and specified event recognition; Perform at least one of the following text parsing operations on the current text data in the current region perception data to obtain the current third behavioral feature of the target object: semantic understanding, syntactic structure analysis, and key entity word extraction; The environmental features of the current area within the target area are extracted from the current area perception data.

9. The method according to any one of claims 1 to 8, characterized in that, Controlling each target device in the group of target devices to be in a device state that matches the environmental characteristics of the predicted area includes: Based on the device status of each target device matching the environmental characteristics of the predicted area, generate corresponding device control instructions for each target device; The device control command corresponding to each target device is generated and sent to each target device to control each target device to be in a device state that matches the environmental characteristics of the predicted area.

10. A device for controlling the status of equipment, characterized in that, include: The prediction unit is used to acquire the perception data collected by a set of perception sensors in the target area at the current moment, and predict the perception features of the predicted area at the next moment. The predicted area environment features in the predicted area perception features are used to represent the predicted regional environmental state that the target object is expected to be in at the next moment. An execution unit is configured to select a set of candidate devices based on the environmental characteristics of the prediction area, and perform the following executability evaluation operation on each candidate device as the current candidate device to obtain an evaluation result corresponding to each candidate device: If the predicted device state of the current candidate device is correlated with a first type of event in a set of currently occurring target events, the first evaluation result is determined as the evaluation result corresponding to the current candidate device, wherein the first type of event is an event that does not allow correlated device states to occur, and the first evaluation result is used to indicate that the device state adjustment operation is not allowed; If the predicted device state of the current candidate device is correlated with a second type of event in the set of target events, a preset value corresponding to the second type of event is subtracted from the initial evaluation value to obtain a target evaluation value, wherein the second type of event is an event that suppresses the occurrence of correlated device states, and the preset value is used to indicate that the second type of event suppresses the occurrence of the predicted device state of the current candidate device. Intensity; if the target evaluation value is greater than or equal to a preset evaluation threshold, a second evaluation result is determined as the evaluation result corresponding to the current candidate device, wherein the second evaluation result is used to indicate that the device state adjustment operation is allowed; if the target evaluation value is less than the preset evaluation threshold, a third evaluation result is determined as the evaluation result corresponding to the current candidate device, wherein the third evaluation result is used to indicate that the device state adjustment operation is not allowed; if the predicted device state of the current candidate device is not associated with any of the target events in the set of target events, a fourth evaluation result is determined as the evaluation result corresponding to the current candidate device, wherein the fourth evaluation result is used to indicate that the device state adjustment operation is allowed; among the set of candidate devices, the candidate device whose corresponding evaluation result indicates that the device state adjustment operation is allowed is determined as the target device, thus obtaining a set of target devices, and controlling each target device in the set of target devices to be in a device state that matches the environmental characteristics of the predicted area.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program, when executed, performs the method of any one of claims 1 to 9.

12. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to execute the method of any one of claims 1 to 9 through the computer program.

Images