Equipment control method and large model training method and device

Abstract

The invention provides an equipment control method and a large model training method and device, relates to the field of artificial intelligence, in particular to the technical fields of large models, natural language processing, deep learning and the like, and can be applied to the application fields of smart home, smart electric appliances, automatic driving and the like. According to the specific implementation scheme, the method comprises the steps of obtaining human body sensing data obtained through image and / or sound collection in a target space; according to the human body sensing data, determining an equipment control strategy matched with the emotion state of the user by adopting a large model; and controlling operation of at least one target electronic device in the target space according to the device control strategy.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, specifically to large models, natural language processing, deep learning and other technical fields, which can be used in application fields such as smart homes, smart appliances, and autonomous driving, and in particular to a device control method, a large model training method and a device. Background Art

[0002] At present, the control of smart devices in scenarios such as smart homes and smart cockpits mostly relies on manual control by users or pre-set rules (also known as preset rules), which provide a certain degree of automated control. Manual control refers to users manually controlling related devices through mobile phone applications or voice assistants, such as adjusting light brightness, changing temperature, playing music, etc. Preset rules refer to users automatically performing certain operations by pre-setting schedules or conditional triggers.

[0003] With the development of artificial intelligence technology, it is very important to fully apply artificial intelligence technology to scenarios such as smart homes and smart cockpits to enhance the intelligence of automation systems, thereby better serving users and meeting their needs and expectations. Summary of the invention

[0004] The present invention provides a device control method, a large model training method and a device.

[0005] According to a first aspect of the present disclosure, a device control method is provided, comprising: acquiring human perception data obtained by collecting images and / or sounds in a target space; determining, based on the human perception data, a device control strategy that matches a user's emotional state using a large model; and controlling the operation of at least one target electronic device in the target space based on the device control strategy.

[0006] According to a second aspect of the present disclosure, a large model training method is provided, including: acquiring human perception data obtained by collecting images and / or sounds in a target space; using a large model to determine a first device control strategy that matches a user's emotional state based on the human perception data; acquiring feedback information of the first device control strategy when controlling device operation based on the first device control strategy; and training the large model based on the feedback information.

[0007] According to a third aspect of the present disclosure, a device control apparatus is provided, comprising: a first acquisition module, used to acquire human perception data obtained by collecting images and / or sounds in a target space; a determination module, used to determine a device control strategy that matches a user's emotional state based on the human perception data using a large model; and a control module, used to control the operation of at least one target electronic device in the target space based on the device control strategy.

[0008] According to a fourth aspect of the present disclosure, a large model training device is provided, including: a first acquisition module, used to acquire human perception data collected by image and / or sound in a target space; a determination module, used to determine a first device control strategy that matches the user's emotional state using a large model based on the human perception data; a second acquisition module, used to acquire feedback information of the first device control strategy when controlling the operation of the device according to the first device control strategy; and a training module, used to train the large model based on the feedback information.

[0009] According to a fifth aspect of the present disclosure, there is provided an electronic device, including:

[0010] at least one processor; and

[0011] a memory communicatively connected to the at least one processor; wherein,

[0012] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the device control method as described in the first aspect, or execute the large model training method as described in the second aspect.

[0013] According to the sixth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the device control method as described in the first aspect, or to execute the large model training method as described in the second aspect.

[0014] According to a seventh aspect of the present disclosure, a computer program product is provided, comprising computer instructions, which, when executed by a processor, implement the device control method as described in the first aspect, or implement the large model training method as described in the second aspect.

[0015] The present disclosure provides a device control method, a large model training method and a device, which have the following beneficial effects:

[0016] Acquire human perception data obtained by collecting images and / or sounds in the target space; determine a device control strategy that matches the user's emotional state using a large model based on the human perception data; and control the operation of at least one target electronic device in the target space based on the device control strategy. The present disclosure can automatically recognize the user's emotional state in the human perception data through a large model, and automatically adjust the device control strategy to respond to the user's emotional state and needs, thereby improving the user's experience.

[0017] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0019] Figure 1 is a flow chart of a device control method provided according to an embodiment of the present disclosure;

[0020] Figure 2 is a flow chart of a device control method provided according to another embodiment of the present disclosure;

[0021] Figure 3 is a flow chart of a device control method provided according to another embodiment of the present disclosure;

[0022] Figure 4 is a flow chart of a device control method provided according to another embodiment of the present disclosure;

[0023] Figure 5 is a flow chart of a device control method provided according to another embodiment of the present disclosure;

[0024] Figure 6 It is a flowchart of a large model training method provided according to an embodiment of the present disclosure;

[0025] Figure 7 is a schematic diagram of the interaction between modules of the context-aware automation system based on a large model provided by the present disclosure;

[0026] Figure 8 A schematic diagram of the structure of a device control device provided according to an embodiment of the present disclosure;

[0027] Fig. 9 is a structural schematic diagram of a large model training device provided according to an embodiment of the present disclosure;

[0028] Fig.10 It is a block diagram of an electronic device used to implement the device control method or large model training method or question-answering processing method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0030] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information are all carried out with the user's consent, and comply with the relevant laws and regulations and do not violate public order and good morals.

[0031] At present, the smart home system has the following main shortcomings:

[0032] 1. Lack of emotional perception

[0033] It is impossible to detect the user's emotional state in real time, making it difficult to adjust the environment according to the user's emotional needs.

[0034] 2. Situational awareness is not comprehensive enough

[0035] Existing systems usually rely on only a single or a few sensors and fail to fully integrate multiple situational factors (such as weather, time, user emotions, etc.) to dynamically adjust home devices.

[0036] 3. Low degree of automation

[0037] Although there are some systems that support preset scenes or time triggers, these operating technologies still require active participation and settings from users, making it difficult to achieve true intelligence and personalization.

[0038] 4. Lack of adaptive learning ability

[0039] The inability to learn and optimize based on users’ long-term behaviors and preferences results in a limited level of intelligence in the system.

[0040] 5. Insufficient multi-user support

[0041] It is difficult to identify different users and automatically adapt to their personalized needs, especially in a home environment where the experience is less than ideal when multiple users share the device.

[0042] In response to at least one of the above problems, the embodiments of the present disclosure provide a device control method, a large model training method and a device.

[0043] The following describes the device control method, large model training method and device of the embodiments of the present disclosure with reference to the accompanying drawings.

[0044] Among them, it should be noted that the executor of the device control method of this embodiment is the device control device, and the executor of the large model training method is the large model training device. The device control device or the large model training device can be implemented by software and / or hardware and can be configured in an electronic device. The electronic device may include but is not limited to a terminal, a server, etc.

[0045] Figure 1 It is a flowchart of a device control method provided according to an embodiment of the present disclosure.

[0046] like Figure 1 As shown, the device control method includes:

[0047] Step 101, obtaining human perception data obtained by collecting images and / or sounds in a target space.

[0048] The target space may refer to any space equipped with electronic devices, such as a home space, a car cabin space, etc., and the present disclosure does not impose any restrictions on this.

[0049] Among them, the electronic equipment equipped in the target space may be, for example, air conditioners, stereos, televisions, electric clothes drying racks, electric curtains, electric fans, etc., and the present disclosure does not impose any restrictions on this.

[0050] In an embodiment of the present disclosure, human perception data obtained by collecting images and / or sounds in a target space may be acquired, wherein the human perception data may include image data and / or voice data, etc., and the present disclosure does not impose any limitation on this.

[0051] As an example, in response to a user's triggering operation, such as entering a room or manually adjusting a device, an image sensor is used to capture images, and / or a sound sensor is used to capture sounds to obtain human perception data.

[0052] It should be noted that in the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved have been subject to the consent of the user, are in compliance with the relevant laws and regulations, and do not violate public order and good morals.

[0053] Step 102: Based on the human perception data, a large model is used to determine a device control strategy that matches the user's emotional state.

[0054] Among them, the user's emotional state can be a real or intuitive description of the user's emotional experience and mental state, such as basic emotions such as joy, anger, sadness, happiness, surprise, fear, love, and hate, or it can also be a positive, negative or neutral emotional tendency.

[0055] It should be noted that the user emotional state may indicate the emotional state of any user in the target space, or may also be the emotional state of the target user in the target space, and the present disclosure does not impose any limitation on this.

[0056] In the disclosed embodiment, human perception data may be input into a large model to obtain a device control strategy determined by the large model, wherein the device control strategy matches the user's emotional state.

[0057] As an example, when the human perception data includes face image data, a large model (LanguageModel) can be used to analyze the user's facial expressions in the face image data to determine the user's emotional state in the target space, and then generate corresponding equipment control strategies based on the user's emotional state; when the human perception data includes voice data, a large model can be used to identify the voice emotions in the voice data to determine the user's emotional state in the target space, and then generate corresponding equipment control strategies based on the user's emotional state.

[0058] For example, when the big model determines that the user's emotional state is negative, the device control strategy is "brighten the indoor lights and play music", "turn on the air conditioner to lower the temperature and reduce the volume", etc.

[0059] Step 103: Control the operation of at least one target electronic device in the target space according to the device control strategy.

[0060] The target electronic device may be one or more electronic devices equipped in the target space, and the present disclosure does not impose any limitation on this.

[0061] In some embodiments, at least one target electronic device in the target space may be controlled to operate according to a device control strategy.

[0062] The device control method of the disclosed embodiment obtains human perception data obtained by collecting images and / or sounds in the target space; according to the human perception data, a large model is used to determine a device control strategy that matches the user's emotional state; according to the device control strategy, the operation of at least one target electronic device in the target space is controlled. In this way, it is possible to automatically identify the user's emotional state in the human perception data through the large model, and automatically adjust the device control strategy to respond to the user's emotional state and needs, thereby improving the user's experience.

[0063] Figure 2 It is a flowchart of a device control method provided according to another embodiment of the present disclosure.

[0064] like Figure 2 As shown, the device control method includes:

[0065] Step 201, obtaining human perception data obtained by collecting images and / or sounds in a target space.

[0066] Step 202, obtaining target monitoring data.

[0067] The target monitoring data may include at least one of meteorological data, environmental status data of the target space, target preference information corresponding to the user in the target space, and historical equipment usage data corresponding to the user.

[0068] Among them, meteorological data may include temperature, wind speed, precipitation type (such as light rain, heavy rain, snow, etc.), air pressure, weather conditions (such as sunny, cloudy, overcast) and other weather data, and the present disclosure does not impose any restrictions on this.

[0069] Among them, the environmental status data may include physical environmental data such as temperature, humidity, light intensity, air quality, etc., which is not limited in the present disclosure.

[0070] Among them, the target preference information can be used to indicate the personalized preferences and tendentious choices shown by users in the target space in adjusting the electronic device.

[0071] Among them, the historical equipment usage data is used to indicate the historical usage of the equipment within a recent preset period of time, including manual operation data, equipment operating status, etc., and the present disclosure does not impose any restrictions on this.

[0072] The most recent preset duration refers to the preset duration before the current time.

[0073] The length of the most recent preset duration may be set and adjusted by a technician according to actual application scenarios, and is not specifically limited in this embodiment.

[0074] In some embodiments, when the target monitoring data includes meteorological data, it can be obtained by calling a meteorological service API (Application Programming Interface); when the target monitoring data includes environmental status data of the target space, the environmental status data in the target space can be collected by environmental sensors in the target space (such as temperature sensors, humidity sensors, light sensors, air quality sensors, etc.); when the target monitoring data includes target preference information corresponding to users in the target space, the historical device usage data of users in the target space can be obtained and analyzed to determine the target preference information corresponding to the users in the target space, or the device usage mode pre-configured by the user can be obtained, and the pre-configured device usage mode can be determined as the target preference information corresponding to the user. Therefore, for different target monitoring data, the corresponding data acquisition method is adopted, and the corresponding data can be obtained quickly and accurately.

[0075] It should be noted that after obtaining the target monitoring data, the target monitoring data can be preprocessed, wherein the data preprocessing may include noise reduction, data normalization, format conversion, etc., which is not limited in the present disclosure.

[0076] Step 203: Generate multimodal data based on the human perception data and the target monitoring data.

[0077] In the disclosed embodiments, user emotion information and target monitoring data may be used to construct multimodal data.

[0078] Step 204, input the multimodal data into the large model to obtain the equipment control strategy determined by the large model.

[0079] Among them, the user scenario model can be used as a data model to indicate user behavior and / or needs.

[0080] In the disclosed embodiment, multimodal data may be input into a large model to obtain a user scenario model determined by the large model.

[0081] In some embodiments, when the multimodal data includes human perception data, meteorological data, environmental status data, target preference information corresponding to users in the target space, and historical equipment usage data, the data of any modality in the multimodal data is input into the corresponding encoder of the large model to obtain corresponding encoding features; the obtained multiple encoding features are input into the hidden layer of the large model to obtain multiple intermediate features; the multiple intermediate features are input into the fully connected layer of the large model to obtain a situation model output by the fully connected layer, and a device control strategy is generated based on the situation model.

[0082] Among them, the situational model may include user emotional state, priority demand attributes, environmental suitability attributes, and historical decision-making habit attributes, etc., which are not limited in the present disclosure.

[0083] Among them, the user's emotional state can be inferred based on human body perception data analysis, for example;

[0084] The priority demand attribute may be used to indicate whether the current settings of the device need to be adjusted, such as may be inferred based on weather data and historical device usage data corresponding to users in the target space;

[0085] The environmental suitability attribute can be the degree of user adaptability to the current environment inferred based on the environmental status data;

[0086] The historical decision-making habit attribute may be a user device usage preference inferred based on historical device usage data and target preference information.

[0087] Step 205: Control the operation of at least one target electronic device in the target space according to the device control strategy.

[0088] The specific implementation forms of step 201 and step 205 can refer to the detailed description in other embodiments of the present disclosure, and will not be described in detail here.

[0089] The device control method of the disclosed embodiment obtains target monitoring data; wherein the target monitoring data includes at least one of meteorological data, environmental status data of the target space, target preference information corresponding to the user in the target space, and historical device usage data corresponding to the user; based on human perception data and target monitoring data, multimodal data is generated; the multimodal data is input into the big model to obtain the device control strategy determined by the big model. Thus, it is possible to use data of different modalities as the data source of the big model, so that the big model can determine the device control strategy in more complex situations, that is, the big model can capture the device control strategy that matches the user's emotional state, environmental state, meteorology, personalized needs, etc., and further enhance the user experience.

[0090] Figure 3 It is a flowchart of a device control method provided according to another embodiment of the present disclosure.

[0091] like Figure 3 As shown, the device control method includes:

[0092] Step 301, obtaining human perception data obtained by collecting images and / or sounds in a target space.

[0093] The specific implementation of step 301 may refer to the detailed description in other embodiments of the present disclosure, and will not be described in detail here.

[0094] Step 302: Identify the human body perception data to obtain the identity information of the user in the target space.

[0095] Among them, the identity information can be used to uniquely indicate the corresponding user, for example, it can be a portrait, voiceprint, name, etc., and the present disclosure does not impose any restrictions on this.

[0096] In some embodiments of the present invention, biometric recognition technology may be used to identify human perception data to obtain identity information of users in the target space.

[0097] It should be noted that biometric recognition technology can include face recognition, voiceprint recognition, body shape recognition, etc., and the present disclosure does not limit this.

[0098] Step 303: Based on the identity information of the user in the target space, query the first database to obtain the target preference information corresponding to the user.

[0099] In some embodiments, a correspondence between a user's identity information and the user's preference information can be pre-constructed and stored in a first database. After obtaining the identity information of the user in the target space, the first database can be queried based on the identity information of the user in the target space to obtain the target preference information corresponding to the user.

[0100] It is understandable that there may be multiple users in the target space, and the users' requirements for the use of the equipment may be different. In order to further determine the target preference information, correspondingly, for any user, based on the user's identity information, the second database is queried to determine the use priority corresponding to the user, and based on the use priorities corresponding to multiple users, the target user is determined from multiple users, and then, according to the target user's identity information, the first database is queried to obtain the preference information corresponding to the target user, and the preference information corresponding to the target user is determined as the target preference information. Therefore, by performing user screening on the use priorities of users in the target space, and determining the screened preference information of the target users as the target preference information, the device control strategy determined by the subsequent large model is more user-friendly, which helps to optimize resource allocation and improve user experience and satisfaction.

[0101] The second database may be used to store the corresponding relationship between the user's identity information and the user's usage priority.

[0102] It should be noted that the first database may be the same as the second database, or may be different, and the present disclosure does not impose any limitation on this.

[0103] It should also be noted that a corresponding usage priority can be set for each user in the target space. As a possible implementation method, the corresponding usage priority can be set according to the user's role in the target space. For example, in a scenario where the target space is a family space, the usage priority of the elderly or children can be higher than the usage priority of other family members; for another example, in a scenario where the target space is a car cabin space, the usage priority of passengers can be higher than the usage priority of the driver.

[0104] As another possible implementation manner, a corresponding usage priority may be set according to the frequency of use of the user's device. For example, the higher the frequency of use of the user's device, the higher the user's usage priority.

[0105] It should be explained that the setting of user usage priority in the above example is only exemplary. In practical applications, it can be set according to actual needs, and the present disclosure does not impose any restrictions on this.

[0106] The target user may be determined from the multiple users based on the usage priorities corresponding to the multiple users. For example, the user with the highest usage priority among the multiple users may be determined as the target user.

[0107] Step 304, obtaining meteorological data, environmental status data of the target space and historical equipment usage data corresponding to the user.

[0108] The specific implementation of step 304 may refer to the detailed description in other embodiments of the present disclosure, and will not be described in detail here.

[0109] In some embodiments, a correspondence between a user's identity information and the user's historical device usage data can be pre-constructed and saved. After obtaining the identity information of the user in the target space, the historical device usage data corresponding to the user can be obtained from the saved correspondence based on the identity information of the user in the target space.

[0110] Step 305 , generating multimodal data based on human perception data, target preference information corresponding to the user, meteorological data, environmental status data of the target space, and historical device usage data corresponding to the user.

[0111] Step 306, input the multimodal data into the large model to obtain the equipment control strategy determined by the large model.

[0112] Step 307: Control the operation of at least one target electronic device in the target space according to the device control strategy.

[0113] The specific implementation of steps 305 to 307 may refer to the detailed description in other embodiments of the present disclosure, and will not be described in detail here.

[0114] It is understandable that the user's device usage preferences may change over time. In order to maintain the correctness and accuracy of the user's preference information, in a possible implementation of the present disclosure, after controlling the operation of at least one target electronic device in the target space according to the device control strategy, feedback data of the device control strategy can be obtained; wherein the feedback data may include user adjustment behavior data; extracting user behavior features from the user adjustment behavior data; and updating the corresponding user's target preference information based on the user behavior features. In this way, the user's preference information can be continuously updated and improved, so as to facilitate the subsequent provision of more personalized and accurate services to users, and continuously provide services and control strategies that meet user expectations, thereby improving the user experience.

[0115] In some embodiments, the feedback data may also include the current device operating status.

[0116] For example, after controlling the operation of at least one target electronic device in the target space according to the device control strategy, when user A in the target space is dissatisfied with the current ambient temperature, the temperature of the target electronic device in the cooling mode is adjusted from 26°C to 20°C, or user A needs music and turns on the audio playback mode. Thus, the executor of the present disclosure can obtain the user adjustment behavior data of the device control strategy and extract the user behavior characteristics, thereby updating the target preference information corresponding to user A based on the user behavior characteristics.

[0117] The device control method of the disclosed embodiment obtains the identity information of the user in the target space by identifying the human body perception data; based on the identity information of the user in the target space, the first database is queried to obtain the target preference information corresponding to the user. Thus, for the user in the target space, the identity information of the user can be identified based on the human body perception data, and then based on the identity information, the target preference information corresponding to the user can be quickly and accurately obtained.

[0118] Figure 4 It is a flowchart of a device control method provided according to another embodiment of the present disclosure.

[0119] like Figure 4 As shown, the device control method includes:

[0120] Step 401, obtaining human perception data obtained by collecting images and / or sounds in a target space.

[0121] Step 402: Based on the human perception data, a large model is used to determine a device control strategy that matches the user's emotional state.

[0122] It should be noted that the specific implementation of step 401 to step 402 can refer to the detailed description in other embodiments of the present disclosure, and will not be described in detail here.

[0123] Step 403: Determine the execution action of any target electronic device according to the device control strategy.

[0124] As an example, assuming that the device control strategy is to turn on the air conditioner to lower the temperature, turn on the audio to broadcast the news, and close the curtains, the target electronic devices include the air conditioner, the audio, and the electric curtains. According to the above device control strategy, the execution action of the air conditioner is determined to be turned on and modulated to the cooling mode, the execution action of the audio is turned on and broadcast the news, and the execution action of the electric curtains is closed.

[0125] Step 404: Generate a corresponding first control instruction based on the execution action of the target electronic device.

[0126] Still using the above example for explanation, when it is determined that the execution action of the air conditioner is to turn on and adjust to the cooling mode, the first control instruction generated for the air conditioner is "turn on and adjust to 25°C in the cooling mode, and adjust the wind speed to medium wind speed"; when it is determined that the execution action of the audio is to turn on and broadcast the news, the first control instruction generated for the audio is "turn on and broadcast the news, the volume is medium", and so on.

[0127] It should be noted that the above examples are merely illustrative, and in actual applications, the first control instructions generated for each target electronic device may also be other, and the present disclosure does not impose any limitation on this.

[0128] Step 405: Control the target electronic device according to the first control instruction.

[0129] In some embodiments, any first control instruction can be sent to a corresponding target electronic device through a corresponding communication protocol, thereby controlling the target electronic device so that the target electronic device operates according to the corresponding first control instruction.

[0130] The communication protocols may include WiFi (Wireless Fidelity), Bluetooth, and the like.

[0131] The device control method of the disclosed embodiment determines the execution action of any target electronic device according to the device control strategy; generates a corresponding first control instruction based on the execution action of the target electronic device; and controls the target electronic device according to the first control instruction. Thus, accurate control of the target electronic device can be achieved, so that the device can operate according to the pre-obtained device control strategy.

[0132] Figure 5 It is a flowchart of a device control method provided according to another embodiment of the present disclosure.

[0133] like Figure 5 As shown, the device control method includes:

[0134] Step 501: Acquire human perception data obtained by collecting images and / or sounds in a target space.

[0135] Step 502: Based on the human body perception data, a large model is used to determine a device control strategy that matches the user's emotional state.

[0136] Step 503: Determine the execution action of any target electronic device according to the device control strategy.

[0137] Step 504: Generate a corresponding first control instruction based on the execution action of the target electronic device.

[0138] It should be noted that the specific implementation of steps 501 to 504 can refer to the detailed description in other embodiments of the present disclosure, and will not be described in detail here.

[0139] Step 505: Perform speech recognition on the speech data in the human perception data to obtain corresponding text data.

[0140] In the embodiments of the present disclosure, speech recognition can be performed on the speech data in the human perception data to obtain corresponding text data. For example, a speech conversion network model (such as a speech synthesis (Speech-to-Text, STT) model) can be used to perform speech conversion on the speech data to obtain corresponding text data.

[0141] Step 506: Perform text analysis on the text data to determine whether the text data includes instruction information.

[0142] In the embodiment of the present disclosure, natural language processing and analysis may be performed on the text data to determine whether the text data includes instruction information.

[0143] The instruction information may indicate an action or operation to be performed.

[0144] In some embodiments, in order to quickly and accurately determine whether the text data includes instruction information, the text data can be subjected to natural language processing and analysis based on the speech data using a language model in a large model to determine whether the text data includes instruction information.

[0145] Step 507: When the text information includes instruction information, convert the instruction information to obtain a second control instruction for controlling the operation of the device.

[0146] In some embodiments, when the text information includes instruction information, the corresponding target electronic device is determined according to the instruction information, and the current operating state information of the corresponding target electronic device is obtained, and then the second control instruction for device operation control is generated according to the current operating state information of the corresponding target electronic device and the instruction information. Thus, the accuracy of the generated second control instruction is improved by combining the current operating state information of the target electronic device pointed to by the instruction information.

[0147] As an example, assuming that the instruction information included in the text information is "turn up the volume of the speaker", the corresponding electronic device can be determined as a speaker based on the instruction information, and the current operating status information of the speaker can be obtained. Assuming that the current operating status information of the speaker includes "the volume is 12", based on the current operating status information of the speaker and the instruction information, a second control instruction "increase the speaker volume to 14" is generated for device operation control.

[0148] Step 508: Generate a third control instruction for controlling the operation of the device according to the first control instruction and the second control instruction corresponding to the at least one target electronic device.

[0149] As an example, assuming that the first control instructions corresponding to at least one target electronic device include "turn on air conditioning and heating mode" and "turn on audio and song playing mode", and the second control instruction is "close the curtains", then, based on the above first control instructions and second control instructions, a third control instruction for device operation control is generated, namely "turn on air conditioning and heating mode, turn on audio and song playing mode, and close the curtains".

[0150] Step 509: Send the third control instruction to the target electronic device.

[0151] In the present disclosure, the third control instruction is sent to the target electronic device so that the target electronic device operates according to the third control instruction.

[0152] It is understandable that the text data may not include instruction information. Therefore, in some embodiments, when the text data does not include instruction information, only the first control instruction needs to be sent to the target electronic device. Thus, in the case where the text data does not include instruction information, only the first control instruction needs to be relied upon to achieve control of the target electronic device.

[0153] In other embodiments, in response to a trigger operation of the user on the interactive interface, a fourth control instruction for controlling the operation of the device can be obtained, and the fourth control instruction can be sent to the corresponding electronic device. Thus, remote or local control of the corresponding electronic device can be achieved through interaction on the interactive interface.

[0154] Among them, the trigger operation includes click operation, slide operation, drag operation, key operation, voice operation, gesture operation, etc., and the present disclosure does not impose any restrictions on this.

[0155] The device control method of the disclosed embodiment obtains corresponding text data by performing voice recognition on voice data; performs text analysis on the text data to determine whether the text data includes instruction information; in the case where the text information includes instruction information, converts the instruction information to obtain a second control instruction for device operation control; generates a third control instruction for device operation control according to the first control instruction and the second control instruction corresponding to at least one target electronic device; and sends the third control instruction to the target electronic device. Thus, the electronic devices in the target space can be controlled not only by relying on the control instructions determined by the device control strategy, but also by relying on the control instructions identified in the voice data, so that the control of the electronic devices is more comprehensive and the flexibility is improved.

[0156] Figure 6It is a flowchart of a large model training method provided according to an embodiment of the present disclosure.

[0157] like Figure 6 As shown, the training method of the large model includes:

[0158] Step 601, obtaining human perception data collected by image and / or sound in the target space.

[0159] Step 602: Determine a first device control strategy that matches the user's emotional state using a large model based on the human perception data.

[0160] It should be noted that the specific implementation of step 601 to step 602 can refer to the detailed description in the embodiment of the above-mentioned device control method of the present disclosure, and will not be described in detail here.

[0161] Step 603: Acquire feedback information of the first device control strategy while controlling the operation of the device according to the first device control strategy.

[0162] The feedback information includes the current device operation status and user adjustment behavior data, etc., which is not limited in the present disclosure.

[0163] Step 604: train the large model based on the feedback information.

[0164] In some embodiments, the evaluation score of the first device control strategy can be determined based on the feedback information, and then the large model can be fine-tuned based on the evaluation score of the first device control strategy. Thus, the large model can be fine-tuned by means of the evaluation score of the first device control strategy determined for the large model, so that the fine-tuned large model can better adapt to the needs of actual application scenarios, improve the accuracy and effectiveness of the device control strategy, and thus improve the overall performance and user satisfaction of the electronic equipment in the target space.

[0165] In some embodiments, in order to improve the accuracy and adaptability of the first device control strategy determined by the large model, when collecting human perception data, the environmental status data of the target space, meteorological data, the operating status data of at least one electronic device in the target space, and the historical device usage data corresponding to the user are collected; the second device control strategy adopted before collecting the human perception data and the corresponding evaluation score are obtained; the human perception data, the evaluation score of the second device control strategy, the second device control strategy, the environmental data, the meteorological data, the operating status data and the historical device usage data are input into the large model to obtain the first device control strategy. In this way, the multimodal data under complex conditions is fully combined, so that the large model can capture the real needs and expectations of the user, thereby improving the accuracy of the first device control strategy determined by the large model.

[0166] Among them, meteorological data may include temperature, wind speed, precipitation type (such as light rain, heavy rain, snow, etc.), air pressure, weather conditions (such as sunny, cloudy, overcast) and other weather data, and the present disclosure does not impose any restrictions on this.

[0167] Among them, the environmental status data may include physical environmental data such as temperature, humidity, light intensity, air quality, etc., which is not limited in the present disclosure.

[0168] The training method of the large model of the embodiment of the present disclosure obtains human perception data collected by image and / or sound in the target space; according to the human perception data, the large model is used to determine the first device control strategy that matches the user's emotional state; when the device operation is controlled according to the first device control strategy, feedback information of the first device control strategy is obtained; and the large model is trained according to the feedback information. In this way, the large model can be automatically trained based on the feedback information, so that the trained large model can learn and generalize the device control strategy that adapts to various complex user emotional states, thereby improving the user satisfaction of the determined device control strategy.

[0169] In order to clearly understand the present disclosure, Figure 7 The schematic diagram of the interaction between modules of the context-aware automation system based on the large model is shown to exemplarily describe the method of the above embodiment. The context-aware automation system includes a multimodal data acquisition module, a context analysis and decision-making module, an intelligent adjustment and control module, a personalized configuration management module, an environmental feedback and adaptive learning module, and a multimodal interaction and control module. Specifically:

[0170] 1. Multimodal data acquisition module

[0171] The multimodal data acquisition module is used to collect multimodal data, wherein the multimodal data includes human perception data, environmental status data, meteorological data, user historical operation records (recorded as historical device usage data in this disclosure), etc., and this disclosure does not impose any restrictions on this.

[0172] 1.1 Human body perception data can be obtained, wherein, for example, image data or video stream data in the human body perception data can be obtained through image acquisition; and / or voice stream data (also called voice data) in the human body perception data can be obtained through sound acquisition;

[0173] 1.2 Use various environmental sensors to monitor temperature, light intensity, air quality, etc. in real time to obtain environmental status data;

[0174] 1.3 Obtain weather data in real time by calling the weather API.

[0175] It should be noted that after acquiring the multimodal data, the above data can be preprocessed, wherein the preprocessing includes noise reduction processing, smoothing processing, data normalization, format conversion, etc., and the present disclosure does not impose any restrictions on this.

[0176] 2. Scenario Analysis and Decision-making Module

[0177] The scenario analysis and decision-making module is used to perform scenario reasoning and analysis based on multimodal data using a large model to obtain a scenario model, and then generate equipment control strategies based on the scenario model.

[0178] The large model includes an encoder layer, a hidden layer, and a fully connected layer, where:

[0179] 2.1 Encoder Layer

[0180] Among them, the encoder layer can include encoders of multiple modes. When the multimodal data is input into the large model, the high-dimensional features of the data of the corresponding mode are extracted through the encoder corresponding to the data of any mode in the multimodal data for subsequent processing and scenario reasoning. Among them, the encoder layer can use convolutional neural network (CNN) to process image data in multimodal data, and recurrent neural network (RNN) or Transformer network to process sequence data (such as voice stream data, video stream data, etc.).

[0181] 2.2 Hidden Layer

[0182] The hidden layer is located after the encoder layer, and multiple layers can be stacked to further abstract and combine the features of the upper layer to generate more complex and nonlinear intermediate features. Through multiple hidden layers, the system can learn more detailed user emotional states, user behavior characteristics, and the deep connection between the current scenario and past history.

[0183] 2.3 Fully Connected Layer

[0184] Among them, the fully connected layer is located at the end of the large model, which can integrate the multi-layer intermediate features output by the hidden layer to generate a situational model. That is, the fully connected layer integrates the contextual situation into a situational model with semantic meaning and description ability, that is, the user's individualized situational model.

[0185] The situation model can include the following four core structural attributes:

[0186] Emotional state attribute: used to indicate the user's current emotional state;

[0187] Priority demand attribute: used to evaluate the priority of users' needs for the current environment;

[0188] Environmental suitability attribute: used to evaluate the user's adaptability or status to the current environmental status;

[0189] Historical decision-making habit attribute: used to indicate the user's operation preference and / or general device usage habits.

[0190] It should be noted that, in practical applications, the context model may also include other structured attributes, which is not limited in the present disclosure.

[0191] It should also be noted that after the situation model is obtained, a device control strategy can be generated based on the situation model.

[0192] 3. Intelligent adjustment and control module

[0193] The intelligent adjustment and control module is used to receive the device control strategy and generate a first control instruction for any target electronic device according to the device control strategy, and then send any first control instruction to the corresponding target electronic device through the corresponding communication protocol, thereby controlling the target electronic device so that the target electronic device operates according to the corresponding first control instruction.

[0194] At the same time, the intelligent adjustment and control module can also obtain and report device feedback data (recorded as device operation status data in this disclosure).

[0195] 4. Personalized configuration management module

[0196] The personalized configuration management module is used to manage and store the adjustment operation records of different users in different environments (recorded as user adjustment behavior data in this disclosure).

[0197] Optionally, the personalized configuration management module can also be used to identify the user's identity information through biometric technology (such as facial recognition and voice recognition, etc.), and based on the user's identity information, query a matching user preference map (referred to as preference information in this disclosure) from a user preference database (referred to as the first database in this disclosure); then, based on the actual recorded user adjustment operation records, update the user's corresponding user preference map.

[0198] 5. Environmental feedback and adaptive learning module

[0199] The environmental feedback and adaptive learning module is used to obtain feedback information, where the feedback information includes user adjustment behavior data and equipment operation status data, and based on the feedback information, continuously optimize the output equipment control strategy through reinforcement learning.

[0200] Among them, in the process of using reinforcement learning technology to continuously optimize the system, state and action are the two cores of reinforcement learning.

[0201] 5.1 Status:

[0202] The state is a description of all key information of the system at the current moment, and the state can include input data from multiple aspects, such as:

[0203] 5.1.1 Environmental status, including environmental status data monitored by multiple environmental sensors, meteorological data obtained by calling the meteorological API, and the current operating status data of the equipment.

[0204] 5.1.2 User status, including current user adjustment behavior data and user emotion information (such as anxiety, pleasure, sleepiness, etc.) obtained through sentiment analysis of human perception data.

[0205] 5.1.3 System status, including the device control strategy currently executed by the system and its Q-Value (referred to as value score in this disclosure), etc.

[0206] 5.2 Action

[0207] An action is an operation that the system takes in a certain state, and this operation will change the state of the system. Actions can include the following:

[0208] 5.2.1 Physical environment actions are used to indicate behavioral actions that directly control the equipment, such as increasing the temperature of the air conditioner, adjusting the brightness of the indoor lighting, adjusting the operating mode and speed of the air purifier, etc.

[0209] 5.2.2 Interaction actions are used to indicate interaction behaviors with users, such as reminding users of the comfort level of the current environment by displaying or hiding, sending prompt information, etc.

[0210] As an example, let’s take a smart home air conditioning system as an example of a context-aware automation system based on a big model. The state may include the following data points:

[0211] Indoor temperature: 28℃

[0212] Set temperature: 24℃

[0213] Outdoor temperature: 35℃

[0214] Indoor humidity: 60%

[0215] User emotional state: Comfortable

[0216] Corresponding to the above states, actions may include the following instructions:

[0217] Adjust the set temperature to 22°C

[0218] Turn on the air conditioner to enhance the dehumidification function

[0219] Prompt user: "It is very hot outside, it is recommended to close the windows"

[0220] In response to user-triggered behavior (such as entering a room, manually adjusting a device, etc.), the multimodal data acquisition module is activated and begins to collect human perception data, environmental status data, meteorological data, etc.

[0221] By continuously cycling between states and actions, the system is able to optimize and adjust equipment control strategies and continuously optimize environmental adaptability and user satisfaction.

[0222] 6. Multimodal interaction and control module

[0223] This module integrates voice interaction with the user interface of the mobile application (referred to as the interactive interface in this disclosure), and in response to the user's triggering operation on the user interface, can obtain control instructions for device operation control and update the display content of the user interface;

[0224] The module can also receive the user's voice data, perform voice recognition on the received voice data, obtain corresponding text data, and perform natural language processing analysis on the text data to determine whether the text data includes instruction information; if the text data includes instruction information, the instruction information is translated into control instructions, and the control instructions are sent to the intelligent adjustment and control module by calling the API.

[0225] Based on the above functional description of each module in the situational awareness automation system, the interaction process between modules in the situational awareness automation system is as follows:

[0226] 1. User interaction with the context-aware automation system: In response to the user's triggering behavior (such as entering a room, manually adjusting a device, etc.), the multimodal data collection module is activated and multimodal data is collected;

[0227] 2. The multimodal data acquisition module processes the data and transmits the processed multimodal data to the scenario analysis and decision-making module;

[0228] 3. The scenario analysis and decision-making module generates scenario models and equipment control strategies:

[0229] After receiving the processed multimodal data, the scenario analysis and decision-making module performs scenario reasoning and analysis through the big model to generate a scenario model, and then generates a device control strategy, and transmits the device control strategy to the intelligent regulation and control module;

[0230] 4. The personalized configuration management module provides user personalized configuration data (referred to as preference information in this disclosure) and historical device usage data:

[0231] The personalized configuration management module transmits the user's personalized configuration data and historical equipment usage data to the situation analysis and decision-making module to influence the final policy output, and transmits the personalized configuration data to the intelligent regulation and control module;

[0232] 5. Intelligent regulation and control module regulates equipment according to equipment control strategy:

[0233] According to the device control strategy generated by the scenario analysis and decision module, the intelligent adjustment and control module generates a control instruction, and sends the device control instruction to the corresponding electronic device so that the electronic device operates according to the corresponding control instruction;

[0234] 6. Environmental feedback and adaptive learning module processes feedback information and optimizes the system

[0235] The environmental feedback and adaptive learning module continuously optimizes the equipment control strategy through reinforcement learning based on the acquired user adjustment behavior data and equipment operation status data, and feeds back the optimization results to the scenario analysis and decision-making module to further improve the system.

[0236] The large model-based situational awareness automation system disclosed in the present invention can at least embody the following advantages:

[0237] 1. Multimodal data integration and complex situation reasoning

[0238] This paper integrates multimodal data sources (such as images, voice, sensor data, meteorological data, etc.) into a unified big model to achieve accurate analysis and reasoning in complex situations. The big model can simultaneously process text, audio, video and environmental sensor data, capture the subtle differences in user emotional states, environmental variables and historical usage patterns, and thus create a comprehensive user context model.

[0239] 2. Contextual awareness and personalized decision making

[0240] By utilizing the context-aware function in the large model, the present invention can automatically identify the optimal decision-making logic in different scenarios in various complex situations, and can learn the complex relationship between the global and local from historical data to generate a high-dimensional user-individualized situational model, greatly enhancing the ability of personalized services.

[0241] 3. Efficient emotional and environmental interactive decision-making

[0242] Integrating the results of sentiment analysis into the system's behavioral reasoning and execution decision modules not only greatly enhances the system's adaptive capabilities, but also allows for dynamic adjustments in real time based on user needs; for example, when the system senses that the user is in a bad mood, it may choose to reduce the ambient light intensity, play soothing music, etc., improving the overall emotional perception of the environment and user participation. This improvement in decision-making intelligence brings a more humane and warm user experience.

[0243] 4. Real-time adaptive optimization and feedback learning

[0244] Combining large models with reinforcement learning can greatly improve the system's adaptive capabilities in uncertain or dynamic situations. The system can not only identify the best control strategy, but also dynamically adjust future decision-making processes based on actual feedback. During the long-term interaction between users and the system, this real-time adaptation and feedback learning effectively improves the operation and control effects and significantly improves the user experience.

[0245] 5. Personalized behavior and predictive modeling

[0246] The system uses a large model to model the user's behavioral characteristics, predict behavior based on historical operation data, and perform personalized optimization, so that the system can adapt to the user's potential preferences and needs in advance at the execution level. For example, based on the user's living habits, emotional expressions, and daily usage methods, the system can

[0247] Automatically adjust environmental settings at the appropriate time, handle abnormal situations in advance, or push precise user services, thereby providing users with forward-looking and humanized full-scenario services.

[0248] In summary, the large-model-based situational awareness automation system disclosed in this disclosure uses a variety of technical means, such as facial expression recognition, voice emotion analysis, large-model algorithms, IoT device control technology, and machine learning, to collect multimodal data such as user emotional state, environmental state data, and meteorological information in real time, infer the user's current needs, and make device control strategies that meet the user's needs, thereby providing a more comfortable life experience. The entire system consists of multiple functional modules, which collaborate to achieve perception, analysis, control and other processes through data transmission and function calls, thereby realizing a new smart experience for smart homes.

[0249] With the above Figures 1 to 5 Corresponding to the device control method provided in the embodiment, the present disclosure also provides a device control apparatus. Figures 1 to 6 The device control method provided in the embodiment corresponds to the device control method, so the implementation of the device control method is also applicable to the device control device provided in the embodiment of the present disclosure, and will not be described in detail in the embodiment of the present disclosure.

[0250] Figure 8 It is a structural schematic diagram of a device control apparatus provided according to an embodiment of the present disclosure.

[0251] like Figure 8 As shown, the device control apparatus 800 may include: a first acquisition module 810, a determination module 820 and a control module 830; wherein:

[0252] The first acquisition module 810 is used to acquire human perception data obtained by collecting images and / or sounds in the target space.

[0253] The determination module 820 is used to determine the device control strategy that matches the user's emotional state based on the human perception data using a large model.

[0254] The control module 830 is used to control the operation of at least one target electronic device in the target space according to the device control strategy.

[0255] In a possible implementation of the embodiment of the present disclosure, the determination module 820 is used to: obtain target monitoring data; wherein the target monitoring data includes at least one of meteorological data, environmental status data of the target space, target preference information corresponding to a user in the target space, and historical equipment usage data corresponding to the user; generate multimodal data based on human perception data and target monitoring data; input the multimodal data into a large model to obtain an equipment control strategy determined by the large model.

[0256] In a possible implementation of the embodiment of the present disclosure, the determination module 820 is used to: identify human perception data to obtain identity information of users in the target space; based on the identity information of users in the target space, query the first database to obtain target preference information corresponding to the user.

[0257] In a possible implementation of the embodiment of the present disclosure, the device control apparatus 800 may further include:

[0258] The second acquisition module is used to acquire feedback data of the device control strategy; wherein the feedback data includes user adjustment behavior data.

[0259] The extraction module is used to extract user behavior features from user adjustment behavior data.

[0260] The updating module is used to update the target preference information of the corresponding user based on the user behavior characteristics.

[0261] In a possible implementation of the embodiment of the present disclosure, there are multiple users, and the determination module 820 is used to: for any user, based on the user's identity information, query the second database to determine the usage priority corresponding to the user; based on the usage priorities corresponding to the multiple users, determine the target user from the multiple users; query the first database according to the identity information of the target user to obtain the preference information corresponding to the target user; and determine the preference information corresponding to the target user as the target preference information.

[0262] In a possible implementation of the embodiment of the present disclosure, the control module 830 is used to: determine the execution action of any target electronic device according to the device control strategy; generate a corresponding first control instruction based on the execution action of the target electronic device; and control the target electronic device according to the first control instruction.

[0263] In a possible implementation of the embodiments of the present disclosure, the human perception data includes voice data; the control module 830 is used to: perform voice recognition on the voice data to obtain corresponding text data; perform text analysis on the text data to determine whether the text data includes instruction information; when the text information includes instruction information, convert the instruction information to obtain a second control instruction for device operation control; generate a third control instruction for device operation control based on the first control instruction and the second control instruction corresponding to at least one target electronic device; and send the third control instruction to the target electronic device.

[0264] In a possible implementation of the embodiment of the present disclosure, the device control apparatus 800 may further include:

[0265] The first sending module is used to send the first control instruction to the target electronic device when the text data does not include instruction information.

[0266] In a possible implementation of the embodiment of the present disclosure, the device control apparatus 800 may further include:

[0267] The third acquisition module is used to acquire a fourth control instruction for device operation control in response to a trigger operation on the interactive interface.

[0268] The second sending module is used to send the fourth control instruction to the corresponding electronic device.

[0269] The device control strategy of the embodiment of the present disclosure obtains human perception data obtained by collecting images and / or sounds in the target space; according to the human perception data, a large model is used to determine a device control strategy that matches the user's emotional state; according to the device control strategy, the operation of at least one target electronic device in the target space is controlled. In this way, it is possible to automatically recognize the user's emotional state in the human perception data through the large model, and automatically adjust the device control strategy to respond to the user's emotional state and needs, thereby improving the user's experience.

[0270] With the above Figure 6 Corresponding to the training method of the large model provided in the embodiment, the present disclosure also provides a training device for a large model. Figure 6 The training method of the large model provided in the embodiment corresponds to the training method of the large model, so the implementation method of the training method of the large model is also applicable to the training device of the large model provided in the embodiment of the present disclosure, and will not be described in detail in the embodiment of the present disclosure.

[0271] Fig. 9 It is a structural schematic diagram of a large model training device provided according to an embodiment of the present disclosure.

[0272] like Fig. 9 As shown, the large model training device 900 may include

[0273] The first acquisition module 910 is used to acquire human perception data collected by performing image and / or sound acquisition in the target space.

[0274] The determination module 920 is used to determine a first device control strategy that matches the user's emotional state using a large model based on the human perception data.

[0275] The second acquisition module 930 is used to acquire feedback information of the first device control strategy when the device operation is controlled according to the first device control strategy.

[0276] The training module 940 is used to train the large model according to the feedback information.

[0277] In a possible implementation of the embodiment of the present disclosure, the training module 940 may be used to: determine the evaluation score of the first device control strategy based on feedback information; and fine-tune the large model based on the evaluation score of the first device control strategy.

[0278] In a possible implementation of the embodiment of the present disclosure, the determination module 920 is used to: when collecting human perception data, collect environmental status data, meteorological data, operating status data of at least one electronic device in the target space, and historical device usage data corresponding to the user; obtain the second device control strategy adopted before collecting the human perception data and the corresponding evaluation score; input the user's emotional state, the evaluation score of the second device control strategy, the second device control strategy, the environmental status data, meteorological data, operating status data and historical device usage data into the large model to obtain the first device control strategy.

[0279] The training device of the large model of the embodiment of the present disclosure obtains human perception data collected by images and / or sounds in the target space; according to the human perception data, the large model is used to determine the first device control strategy that matches the user's emotional state; when the device operation is controlled according to the first device control strategy, feedback information of the first device control strategy is obtained; and the large model is trained according to the feedback information. Thus, the large model can be automatically trained based on the feedback information, so that the trained large model can learn and generalize the device control strategy that adapts to various complex user emotional states, thereby improving the user satisfaction of the determined device control strategy.

[0280] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0281] Fig.10 A schematic block diagram of an example electronic device 1000 that can be used to implement an embodiment of the present disclosure is shown. The electronic device 1000 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0282] like Fig.10As shown, the electronic device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a ROM (Read-Only Memory) 1002 or a computer program loaded from a storage unit 1008 to a RAM (Random Access Memory) 1003. In the RAM 1003, various programs and data required for the operation of the electronic device 1000 can also be stored. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An I / O (Input / Output) interface 1005 is also connected to the bus 1004.

[0283] Multiple components in the electronic device 1000 are connected to the I / O interface 1005, including: an input unit 1006, such as a keyboard, a mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a disk, an optical disk, etc.; and a communication unit 1009, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1009 allows the electronic device 1000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0284] The computing unit 1001 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a CPU (Central Processing Unit), a GPU (Graphic Processing Units), various dedicated AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, DSP (Digital Signal Processor), and any appropriate processor, controller, microcontroller, etc. The computing unit 1001 performs the various methods and processes described above, such as the above-mentioned device control method or the training method of the large model. For example, in some embodiments, the above-mentioned device control method or the training method of the large model may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded into RAM 1003 and executed by the computing unit 1001, one or more steps of the device control method or the large model training method described above may be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to execute the device control method or the large model training method or the question-answering processing method in any other appropriate manner (e.g., by means of firmware).

[0285] Various embodiments of the systems and techniques described above herein may be implemented in digital electronic circuit systems, integrated circuit systems, FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), ASSPs (Application Specific Standard Products), SOCs (System On Chips), CPLDs (Complex Programmable Logic Devices), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs that may be executed and / or interpreted on a programmable system including at least one programmable processor that may be a dedicated or general-purpose programmable processor that may receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0286] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, implements the functions / operations specified in the flow chart and / or block diagram. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0287] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include electrical connections based on one or more lines, portable computer disks, hard disks, RAM, ROM, EPROM (Electrically Programmable Read-Only-Memory) or flash memory, optical fiber, CD-ROM (Compact Disc Read-Only Memory), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0288] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0289] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: LAN (Local Area Network), WAN (Wide Area Network), the Internet, and blockchain networks.

[0290] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0291] It should be noted that artificial intelligence is a discipline that studies how computers can simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.), and includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing; artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, as well as machine learning / deep learning, big data processing technology, knowledge graph technology, and other major directions.

[0292] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0293] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A device control method, comprising: Acquiring human perception data obtained by collecting images and / or sounds in a target space; Determine, based on the human perception data, a device control strategy that matches the user's emotional state using a large model; According to the device control strategy, the operation of at least one target electronic device in the target space is controlled.

2. The method according to claim 1, wherein: The method of using a large model to determine a device control strategy that matches the user's emotional state based on the human body perception data includes: Acquire target monitoring data; wherein the target monitoring data includes at least one of meteorological data, environmental status data of the target space, target preference information corresponding to a user in the target space, and historical device usage data corresponding to the user; Generate multimodal data based on the human perception data and the target monitoring data; The multimodal data is input into the large model to obtain the device control strategy determined by the large model.

3. The method according to claim 2, wherein: The obtaining of target monitoring data includes: Identify the human body perception data to obtain identity information of a user in the target space; Based on the identity information of the user in the target space, the first database is queried to obtain the target preference information corresponding to the user.

4. The method according to claim 3, wherein: The method further comprises: Acquiring feedback data of the device control strategy; wherein the feedback data includes user adjustment behavior data; extracting user behavior features from the user adjustment behavior data; Based on the user behavior characteristics, the target preference information of the corresponding user is updated.

5. The method according to claim 3, wherein: There are multiple users, and querying a first database based on the identity information of the users in the target space to obtain target preference information corresponding to the users includes: For any user, based on the identity information of the user, query the second database to determine the usage priority corresponding to the user; Determining a target user from among the multiple users based on usage priorities corresponding to the multiple users; Querying the first database according to the identity information of the target user to obtain preference information corresponding to the target user; The preference information corresponding to the target user is determined as the target preference information.

6. The method according to claim 1, wherein: The controlling the operation of at least one target electronic device in the target space according to the device control strategy includes: Determining an execution action of any of the target electronic devices according to the device control strategy; Based on the execution action of the target electronic device, generating a corresponding first control instruction; The target electronic device is controlled according to the first control instruction.

7. The method according to claim 6, wherein: The human body perception data includes voice data; and controlling the target electronic device according to the first control instruction includes: Performing speech recognition on the speech data to obtain corresponding text data; Performing text analysis on the text data to determine whether the text data includes instruction information; In the case where the text information includes instruction information, converting the instruction information into a second control instruction for controlling the operation of the device; generating a third control instruction for controlling operation of the device according to the first control instruction and the second control instruction corresponding to the at least one target electronic device; The third control instruction is sent to the target electronic device.

8. The method according to claim 7, wherein: The method further comprises: In a case where the text data does not include the instruction information, the first control instruction is sent to the target electronic device.

9. The method according to any one of claims 1 to 8, wherein: The method further comprises: In response to a trigger operation on the interactive interface, acquiring a fourth control instruction for controlling operation of the device; The fourth control instruction is sent to the corresponding electronic device.

10. A large model training method, comprising: Acquire human perception data collected through image and / or sound in the target space; Determine, based on the human perception data, a first device control strategy matching the user's emotional state using a large model; Under the condition that the operation of the device is controlled according to the first device control strategy, obtaining feedback information of the first device control strategy; The large model is trained according to the feedback information.

11. The method according to claim 10, wherein: The step of training the large model according to the feedback information includes: Determining an evaluation score of the first device control strategy according to the feedback information; The large model is fine-tuned according to the evaluation score of the first device control strategy.

12. The method according to claim 11, characterized in that The method of using a large model to determine a first device control strategy that matches the user's emotional state based on the human body perception data includes: When the human perception data is collected, environmental status data of the target space, meteorological data, operating status data of at least one electronic device in the target space, and historical device usage data corresponding to the user are collected; Obtaining a second device control strategy and a corresponding evaluation score adopted before the human perception data is collected; The human perception data, the evaluation score of the second device control strategy, the second device control strategy, the environmental status data, the meteorological data, the operating status data and the historical device usage data are input into the large model to obtain the first device control strategy.

13. A device control apparatus, comprising: A first acquisition module is used to acquire human perception data obtained by collecting images and / or sounds in a target space; A determination module, configured to determine a device control strategy matching a user's emotional state using a large model based on the human body perception data; A control module is used to control the operation of at least one target electronic device in the target space according to the device control strategy.

14. A large model training device, comprising: A first acquisition module is used to acquire human perception data collected by performing image and / or sound acquisition in a target space; A determination module, configured to determine a first device control strategy matching the user's emotional state using a large model according to the human body perception data; A second acquisition module, configured to acquire feedback information of the first device control strategy when controlling the operation of the device according to the first device control strategy; A training module is used to train the large model according to the feedback information.

15. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in any one of claims 1 to 9, or execute the method described in any one of claims 10 to 12.

16. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to enable the computer to execute the method described in any one of claims 1 to 9, or to execute the method described in any one of claims 10 to 12.

17. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the method according to any one of claims 1 to 9, or implements the method according to any one of claims 10 to 12.

Images