FTTR-based device control method, system, device, medium and program product

Abstract

The present application discloses a device control method, system, device, medium and program product based on FTTR, which belongs to the field of smart home technology. The method includes: obtaining an initial device control model; training the initial device control model through a first loss function and at least one second loss function to obtain a target device control model, wherein the first loss function is used to characterize the degree of matching between the probability value of the device executing multiple candidate actions and the real action label, and at least one second loss function is determined according to a user preference index, and the user preference index includes at least one of a device operation energy consumption index, a device operation safety index and a user perception feedback index; and sending the target device control model to the device. In this way, the problem of deviation in the response of the relevant device control system to fuzzy requests can be effectively solved, and the precise control capability of the device control system can be improved.

Description

Technical Field

[0001] The present application relates to the field of smart home technology, and in particular to a device control method, system, device, medium and program product based on FTTR. Background Art

[0002] With the rapid development of IoT technology, smart home devices are gradually becoming integrated into modern home environments. However, existing technical architectures still have significant limitations, especially at the interaction level. Traditional smart home control systems mostly rely on preset rules or simple voice commands to control home devices. Device behavior patterns are fixed in the initial configuration, which makes the system lack a deep understanding of user needs and lacks accurate response. For example, when a user issues a vague request such as "adjust the living room temperature to a suitable level," the system often cannot accurately grasp the user's specific needs, resulting in a deviation between the service response and the user's actual needs, forcing users to make frequent manual adjustments, affecting the convenience and intelligence level of the smart home system. Summary of the Invention

[0003] The embodiments of the present application provide a device control method, system, device, medium and product based on FTTR to at least solve the problem of deviation in the response of related smart home control systems to fuzzy requests.

[0004] In order to solve the above technical problems, this application is implemented as follows:

[0005] In the first aspect, an embodiment of the present application provides a device control method based on FTTR, including: obtaining an initial device control model, wherein the initial device control model is used to adjust the operating state of the device according to target intention information, and an edge computing module is provided on the device; the initial device control model is trained by a first loss function and at least one second loss function to obtain a target device control model, wherein the first loss function is used to characterize the degree of matching between the probability values ​​of the device performing multiple candidate actions and the real action labels, and the at least one second loss function is determined according to a user preference index, and the user preference index includes at least one of a device operation energy consumption index, a device operation safety index, and a user perception feedback index; and the target device control model is sent to the device.

[0006] In the second aspect, an embodiment of the present application provides a device control method based on FTTR, including: obtaining target intention information for a device; inputting the target intention information into a target device control model to obtain an adjustment instruction for the device, wherein the target device control model is obtained by training an initial device control model through a first loss function and at least one second loss function, the first loss function is used to characterize the degree of matching between the probability values ​​of the device performing multiple candidate actions and the real action labels, and the at least one second loss function is determined according to a user preference index, and the user preference index includes at least one of a device operation energy consumption index, a device operation safety index and a user perception feedback index; the operating state of the device is adjusted through the adjustment instruction.

[0007] In a third aspect, an embodiment of the present application provides an FTTR-based device control system, including a cloud platform and an edge computing device; the cloud platform is used to execute the steps of the method described in the first aspect above, and the edge computing device is used to execute the steps of the method described in the second aspect above.

[0008] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the method described in the first or second aspect above are implemented.

[0009] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first or second aspect above are implemented.

[0010] In a sixth aspect, an embodiment of the present application provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer performs the steps of the method described in the first or second aspect above.

[0011] In an embodiment of the present application, an initial device control model is obtained, which is used to adjust the operating state of the device based on target intent information. An edge computing module is provided on the device. The initial device control model is trained using a first loss function and at least one second loss function to obtain a target device control model. The first loss function is used to characterize the degree of match between the probability values ​​of the device executing multiple candidate actions and the actual action labels. The at least one second loss function is determined based on user preference indicators, which include at least one of a device operating energy consumption indicator, a device operating safety indicator, and a user perception feedback indicator. The target device control model is then sent to the device. In this way, optimizing the initial device control model using the first loss function and the at least one second loss function ensures that the device can accurately execute actions that match the target intent information. Simultaneously, by adjusting the model using user preference indicators (such as energy consumption, safety, and user feedback), the initial device control model fully learns the user's preferences and needs, thereby effectively resolving the problem of biased responses of smart home control systems to fuzzy requests and improving the precise control capabilities of smart home control systems.

[0012] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0014] Figure 1 A schematic diagram of the interaction process of a smart home control system provided by some embodiments of the present application is shown;

[0015] Figure 2 A schematic flow chart of a FTTR-based device control method provided in some embodiments of the present application is shown;

[0016] Figure 3 shows an architecture diagram of a target neural network provided by some embodiments of the present application;

[0017] Figure 4 Schematic diagram showing the flow of FTTR-based device control methods provided in other embodiments of the present application;

[0018] Figure 5 A schematic diagram of the workflow of the interactive system provided in some embodiments of the present application is shown;

[0019] Figure 6 One of the control flow diagrams of the LLM control module provided in some embodiments of the present application is shown;

[0020] Figure 7 FIG2 shows a second control flow diagram of the LLM control module provided in some embodiments of the present application;

[0021] Figure 8 shows an adaptive evolution sequence diagram provided by some embodiments of the present application;

[0022] Figure 9 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0023] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0024] The smart home control system is a home management system based on the Internet and artificial intelligence technology. It realizes automated management and intelligent control of the home environment by integrating various smart devices, sensors, control terminals, etc. Figure 1 A schematic diagram of the interaction flow of a smart home control system provided by some embodiments of the present application is shown. As shown in the figure, home users interact with the smart home control system 110 through voice, text, gestures, and other means, sending commands to the smart home control system 110 and receiving feedback from it. The smart home control system 110 communicates with the device layer within the home via a fiber-to-the-room (FTTR) network 120. The FTTR network 120 provides a high-bandwidth, low-latency communication foundation, ensuring efficient connectivity between devices. Within the home network boundary, edge computing devices 130 perform local data processing to reduce latency and protect privacy. Edge computing devices 130 interact with multiple device modules, including sensors 141, cameras 142, network-attached storage (NAS) 143, and smart devices 144. Sensors 141 collect environmental data; cameras 142 capture video streams; NAS 143 stores home data and logs; and smart devices 144 execute control commands. These device modules work together with the smart home control system 110 through the edge computing device 130 to form a closed-loop control process.

[0025] In response to the problem that most related smart home control systems rely on preset rules or simple voice commands to control home appliances, and there are deviations in the responses to fuzzy requests, an embodiment of the present application provides a device control method based on FTTR. This method optimizes the initial device control model through a first loss function and at least one second loss function, which can ensure that the device can accurately execute actions that match the target intention information. At the same time, the model is adjusted through user preference indicators such as energy consumption, safety, and user feedback, so that the initial device control model can fully learn the user's preferences and needs, so as to solve the problem that the smart home control system has deviations in the response to fuzzy requests.

[0026] See also Figure 2 , Figure 2 A flow chart illustrating a method for controlling an FTTR-based device according to some embodiments of the present application is shown. The method may be executed by a backend server of the cloud platform 150 of the smart home control system. The server may be a standalone server or a server cluster consisting of multiple servers. As shown in the figure, the FTTR-based device control method 200 may include the following steps:

[0027] Step 201: Obtain an initial device control model, which is used to adjust the operating state of the device according to the target intention information. An edge computing module is set on the device.

[0028] Among them, the target intention information in step 201 can be the intention information of adjusting the device operating status corresponding to the user input instruction, for example, "dim the living room lights", or it can be the adjustment intention information determined by the smart home control system 110 based on the multimodal perception data of the current environment; the edge computing module in step 201 is used to locally process the collected sensor data and device status data to obtain multimodal perception data.

[0029] In one exemplary embodiment, the smart home control system 110 sends a model update request to the cloud platform 150. The cloud platform 150 obtains the initial device control model corresponding to the model update request, performs federated learning and model updates on the initial device control model, and returns the updated target device control model to the smart home control system 110. Furthermore, the device is equipped with an edge computing model. The local processing capabilities of this edge computing module are key, reducing reliance on the external cloud platform 150, shortening response time, improving the real-time performance of the system, and enhancing the security of local data.

[0030] Step 202: The initial device control model is trained using a first loss function and at least one second loss function to obtain a target device control model. The first loss function is used to characterize the degree of match between the probability values ​​of the device executing multiple candidate actions and the actual action labels. The at least one second loss function is determined based on a user preference index. The user preference index includes at least one of a device operation energy consumption index, a device operation safety index, and a user perception feedback index.

[0031] In an exemplary embodiment, the cloud platform 150 trains the initial device control model through a first loss function to ensure that the device can perform actions that match the target intention information. At the same time, by introducing at least one second loss function, the initial device control model is trained based on at least one user preference indicator among energy consumption, safety and user feedback, so that the initial device control model fully learns the user's needs and preferences to ensure that the device can perform actions that are more in line with the user's intentions, reduce user manual intervention, and improve user satisfaction.

[0032] Step 203: Send the target device control model to the device.

[0033] Continuing with the above embodiment, the cloud platform 150 sends the trained target device control model to the device, and the target device control model is used to adjust the operating state of the device according to the target intention information.

[0034] Through the above steps, the initial device control model is optimized by the first loss function and at least one second loss function, which can ensure that the device can accurately perform actions that match the target intention information. At the same time, the model is adjusted through user preference indicators (such as energy consumption, safety, user feedback, etc.) so that the initial device control model can fully learn the user's preferences and needs, thereby effectively solving the problem of deviation in the response of the smart home control system to fuzzy requests and improving the precise control capability of the smart home control system.

[0035] In some embodiments, the first loss function in step 202 is also called the device control loss. , which is used to represent the degree of match between the probability values ​​of home devices performing multiple candidate actions and the real action labels.

[0036] In an exemplary embodiment, a cross entropy loss may be used to optimize the action accuracy, and the formula is as follows:

[0037] ;

[0038] in, is the real action label, soft-label encoding, for example, [0,1,0] means "turn on the air conditioner", [0,24,0] means "adjust the air conditioner to 24 degrees", The model is i The predicted probability of each candidate action.

[0039] In some embodiments, the second loss function in the above-mentioned step 202 includes a third loss function corresponding to the energy consumption index of the device operation; the third loss function is also called energy consumption optimization loss, including a first loss term and a second loss term, the first loss term is used to characterize the degree of difference between the predicted power and the actual power when the device performs the candidate action, and the second loss term is used to characterize the degree of difference between the predicted average power and the actual average power of the device based on the comfort estimate compensation within a preset time period, the comfort estimate is determined based on historical energy consumption data and user comfort score, and the user comfort score is determined based on the multimodal perception data in the target intention information.

[0040] In an exemplary embodiment, energy consumption optimization loss Dynamic weighted error and comfort compensation can be used, and the formula is as follows:

[0041] ;

[0042] ;

[0043] The energy consumption optimization loss It includes the first loss term and the second loss term. The first loss term, that is, the first part is the mean square error constraint device energy consumption deviation, which is predicted by comparing the power The actual physical power of the device The loss term is constructed by the difference between , which imposes a quadratic penalty on over-power operation and is suitable for energy consumption control of high-power equipment such as air conditioners and water heaters;

[0044] The second loss term, that is, the latter part is the dynamic threshold L1 loss. Combined with the historical energy consumption data of the equipment, the penalty threshold is dynamically adjusted. The predicted average power for a preset time period, for example, the past 24 hours; The actual average power of the device during the forecast period.

[0045] in, During the peak electricity price period, To control the weight decay rate, The user comfort score (range 0~1) can be calculated based on multimodal perception data such as temperature, light, and noise. It is the comfort compensation coefficient. In specific applications, it can be set to 0.5 to suppress the decrease in comfort caused by excessive energy saving.

[0046] In other embodiments, the second loss function in the above-mentioned step 202 includes a fourth loss function corresponding to the equipment operation safety index; the fourth loss function is also called the safety constraint loss, which is used to characterize the safety level of the target action among multiple candidate actions performed by the equipment.

[0047] In an exemplary embodiment, the safety constraint loss , which is used to suppress dangerous operations, and its formula is as follows:

[0048] ;

[0049] in, Indicates that the status Next, execute the action The safety score ranges from [0,1]. The higher the safety score, the smaller the loss. Represents an exponent greater than or equal to 1, used to adjust the growth curve of the insecurity level. In specific applications, it can be set to 2 to increase the penalty for severe insecurity and ignore mild insecurity.

[0050] In other embodiments, the second loss function in the above-mentioned step 202 includes a fifth loss function corresponding to the user perception feedback index; the fifth loss function is also called comfort loss, including a third loss term, a fourth loss term and a fifth loss term, the third loss term is used to characterize the degree of difference between the real user preference probability distribution and the predicted user preference probability distribution, the real user preference probability is determined based on the adjustment intention information, and the adjustment intention information is the user feedback information after adjusting the operating state of the device based on the target device control model, the fourth loss term is used to characterize the degree of difference between multiple candidate actions and user control behaviors, and the fifth loss term is used to characterize the degree of consistency between the comfort prediction results of the first scene and the comfort prediction results of the second scene, and the difference between the environmental parameters of the first scene and the second scene is less than the preset parameter threshold.

[0051] In an exemplary embodiment, the comfort loss , integrating user feedback, the formula is as follows:

[0052] ;

[0053] ;

[0054] ;

[0055]

[0056] Among them, the third loss item Represents display feedback loss. After adjusting the operating state of the device based on the target device control model, the adjustment intention information is determined through the user's active rating. For example, the control effect is rated from 1 to 5 stars in the application or voice assistant, and the context state, such as temperature and humidity, time period, usage habits, etc., jointly generate a set of multi-category behavior preference probability distribution. , which is used to characterize the user's preference for each control strategy in a specific scenario; the intelligent control system predicts the probability distribution of user preferences in the current state based on sensor input and historical interaction records , ϵ is a smoothing term to prevent numerical instability when the probability is 0;

[0057] The fourth loss item represents the implicit feedback loss, or the behavior preference gradient alignment loss, and indicates the degree of reversal between the system's prediction W (representing the system's prediction of all candidate actions) and the preference gradient G. User implicit feedback refers to the user manually executing the device or secondary voice control coverage model decision. G represents a "behavior preference gradient vector" of the same dimension as W that satisfies the following formula. This gradient vector represents the behavior that the user wants the system to promote or weaken:

[0058] ;

[0059] in, Indicates the action originally performed by the system. Indicates the action the user actually wants to perform through implicit feedback, i Represents the candidate action index.

[0060] Fifth loss item Represents consistency loss, ensuring that the comfort score is consistent in similar environmental scenes, avoiding model oscillation due to noise feedback, and screening similar scene sample pairs through clustering algorithm , It represents the comfort score predicted by the model, wherein the first scene and the second scene whose environmental parameter difference is less than a preset parameter threshold can be determined as similar environmental scenes.

[0061] In some embodiments, in step 202, the initial device control model is trained using the first loss function and the at least one second loss function to obtain the target device control model, including:

[0062] According to each user preference index and the priority of the first loss function, a preset number of target loss functions with higher priority among the multiple loss functions are determined, and the multiple loss functions include the first loss function and the second loss function; according to the preset conditions, the weights of the multiple loss functions are determined, and the preset conditions are that the weight of each target loss function is set within the preset weight range corresponding to each target loss function; according to the multiple loss functions and the weights corresponding to each loss function, the initial device control model is trained to obtain the target device control model.

[0063] In an exemplary embodiment, according to the priority of each user preference index and the first loss function, a preset number of target loss functions with higher priority among the multiple loss functions are determined, wherein the priority of each user preference index and the first loss function can be set according to specific needs, and the number of target loss functions can be one or more, which can be set according to specific needs. For example, the safety constraint loss function can be set as Determine the target loss function; adopt the safety priority dynamic limit weighting method to give priority to the safety constraint loss, and the weight range of the safety loss satisfies: , that is, in each training step, the weight of the safety loss cannot be lower than a certain threshold to ensure that the model always focuses on the safety task. The remaining weights of the remaining losses are dynamically distributed according to the momentum amplitude (BMW):

[0064] ;

[0065] ;

[0066] in, Indicates the j The first-order momentum of the loss term is used to indicate the trend of the loss during training. Indicates the momentum of the current task. At the same time, ensure . is a very small constant to avoid division by zero or numerical instability. The more drastic the change in momentum, the larger the weight of the task, indicating that it is still actively learning.

[0067] In some embodiments, in step 201 above, obtaining the initial device control model includes:

[0068] Acquire sample data, which includes multimodal perception data of the target area, and at least one of user behavior data corresponding to the multimodal perception data and a real action label of the device; train a preset neural network model based on the sample data to obtain an initial device control model.

[0069] Among them, the multimodal perception data in this step may include device status data, sensor data, device usage behavior data and environmental monitoring data; device status data such as switch, mode, energy consumption, command, etc., sensor data such as temperature, humidity, movement, etc., device usage behavior data such as the frequency of use of the device within a preset time period, or the frequency of use of the device, etc., environmental monitoring data such as audio data, video data, etc.; user behavior data in this step may be the user's input instructions, or the user's actions, feedback rating information, etc.; the real action label of the device in this step may be the encoding data of the real action of the device such as [0,1,0], [0,24,0], [0,1,0] means "turn on the air conditioner", [0,24,0] means "turn on means adjust the air conditioner to 24 degrees"; the preset neural network model in this step may be a convolutional neural network, a deep reinforcement learning network, an augmentation dynamic sparse activation network (ADSAN), a long short-term memory network (LSM), a neural network, ... Here, ADSAN is a model that improves the efficiency of neural networks by combining sparse activation and dynamic adjustment mechanisms.

[0070] In one exemplary embodiment, cloud platform 150 acquires sample data and constructs a feedback-optimized sample set based on the multimodal perception data, user behavior data, and device action labels included in the sample data. This feedback-optimized sample set is then used to train a pre-set neural network, enabling the neural network to fully learn the mapping relationship between user behavior data, multimodal environmental perception data, and device control policies for home devices within the target area. The training process can adjust the network parameters of the pre-set neural network using a pre-set loss function to minimize the error between device actions and actual actions, thereby obtaining an initial device control model.

[0071] In this way, by training the preset neural network through sample data, the preset neural network can continuously optimize the device control logic based on multimodal perception data samples, user behavior data and the real action labels of the device. The initial device control model obtained by training can better adapt to the user's personalized needs.

[0072] In some possible implementations, the above-mentioned obtaining of sample data includes:

[0073] Receive sample data sent by the device, wherein the sample data is obtained by the device splicing the original collected data to obtain fused data, constructing a fusion matrix based on the fused data, and processing the fusion matrix according to the cross-attention mechanism.

[0074] In an exemplary embodiment, in order to improve the response speed of the system, improve bandwidth allocation, and protect user privacy, the smart home control system 110 can obtain multimodal perception data of the target area through the edge computing module on the FTTR network 120 control device, determine fusion data based on the multimodal perception data, construct a fusion matrix based on the fusion data, and process the fusion matrix according to the cross-attention mechanism to obtain sample data.

[0075] In this way, the edge computing module on the device processes the multimodal perception data from different perception sources to generate a unified, high-dimensional sample data, which enables the cloud platform 150 to understand and respond to complex information in the environment more accurately. At the same time, it is conducive to improving response speed, improving bandwidth allocation, and ensuring the security of multimodal perception data in the target area.

[0076] In some possible implementations, the preset neural network includes multiple sub-networks corresponding to target functional tasks; the training of the preset neural network model based on the sample data to obtain the initial device control model includes:

[0077] The preset neural network is iteratively trained using sample data. During each iterative training process, the network parameters of each sub-network and the exponential moving average of the loss function gradient corresponding to each sub-network are obtained, and the exponential moving average is used to characterize the sensitivity of the network parameters of the sub-network to the loss function; based on the network parameters and the exponential moving average of each sub-network, the activation mask of each sub-network is determined; a target sub-network whose activation mask meets a preset threshold is selected from multiple sub-networks, the target sub-network is trained using the sample data, and the network parameters of each sub-network and the exponential moving average of the loss function gradient corresponding to each sub-network are updated until the preset convergence conditions are met, and the preset neural network after convergence is determined as the initial device control model.

[0078] In an exemplary embodiment, an enhanced dynamic sparse activation network can be selected as a preset neural network. The preset neural network includes 6 sub-networks, each of which focuses on a specific functional task, as shown in the following table:

[0079] Table 1. Functional tasks of each sub-network in the preset neural network

[0080]

[0081] like Figure 3As shown, the multimodal perception data is preprocessed at the input layer. Specifically, multimodal perception data is obtained, such as images, audio, temperature, humidity, device switches, energy consumption, control commands issued by users, etc., and time-aligned and initialized according to the set time window to obtain multiple time series features; multiple time series features are spliced ​​to obtain fused data, a fusion matrix is ​​constructed based on the fused data, and the fusion matrix is ​​fused and encoded according to the cross-attention mechanism to obtain sample data. In the inference network stage, the target sub-network is selected from the energy consumption optimization network, the safety anomaly network, the user comfort network, the equipment fault prediction network, the environment adaptation network, and the cross-device linkage network, and the control instructions of the device are generated through the target self-network. The selection process of the target sub-network is as follows:

[0082] Get the network parameters of each subnetwork And the exponential moving average g of the loss function gradient corresponding to each sub-network. The exponential moving average gradient value g is used to indicate the sensitivity of the network parameters of the sub-network to the loss function; the sub-network with a larger gradient is usually more critical to the optimization of the current task, indicating that it needs to be activated more frequently to reduce the loss. According to the network parameters of each sub-network and the exponential moving average g, determine the activation mask of each sub-network , select the target subnetwork whose activation mask meets the preset threshold from multiple subnetworks, and generate the device control logic by inputting the joint representation vector into the target subnetwork, for example, dynamically select the activation path to retain only The top-2 subnetworks in the network, for example, the energy consumption prediction module and the security decision module, and the rest of the paths are set to zero.

[0083] Train the target sub-network through sample data and update the network parameters of each sub-network The exponential moving average g of the loss function gradient corresponding to each sub-network is calculated until the preset convergence condition is met, and the preset neural network after convergence is determined as the initial device control model.

[0084] In some possible implementations, determining the activation mask of each sub-network based on the network parameters and the exponential moving average of each sub-network includes:

[0085] The network parameters of each sub-network and the exponential moving average gradient value are multiplied element by element to obtain the target product value; the absolute value of the target product value is averaged to obtain the parameter influence estimate of each sub-network, which is used to indicate the degree of influence of the sub-network on the network parameter update during training; the parameter influence estimate is normalized to obtain the activation probability; the activation probability of each sub-network is sampled to obtain the activation mask of each sub-network.

[0086] In an exemplary embodiment, the sparse gating module generates sub-network activation probabilities by driving the input feature uncertainty. First, the statistical information of the parameter derivatives of each sub-network is extracted:

[0087] ;

[0088] in, For the i The network parameters of the sub-network, g is the exponential moving average gradient value, The Exponential Moving Average (EMA) gradient retained from the previous step, is element-wise multiplication.

[0089] After obtaining the statistical information of the derivative, the activation probability is modeled and the parameter influence is estimated Mapped to activation probability, normalized using the sigmoid function:

[0090] ;

[0091] in, is a learnable temperature control factor that controls the mapping slope, Represents the learnable activation threshold. Subnetworks with high parameter influence estimates have higher activation probabilities after sigmoid mapping. After obtaining the above parameters, we use Binary Concrete distribution to sample the activation mask from the probability:

[0092] ;

[0093] Among them, u ~ Uniform(0,1), Indicates temperature hyperparameter, the smaller it is, the closer it is to hard binary.

[0094] To further reduce the computational cost of the inference phase while ensuring the model's responsiveness to key inputs, the present embodiment makes targeted improvements to the sparse gated activation mechanism, focusing on optimizing the learning method and distribution characteristics of its activation probability, thereby significantly improving the system's performance in inference tasks such as device control. Compared to conventional sparse methods, the average sparsity rate is increased by 35.2%: without reducing accuracy, the average number of activated neurons in the inference phase decreases by 35.2%, significantly reducing computing resource requirements.

[0095] See also Figure 4 , Figure 4The following is a flow chart illustrating a method for controlling an FTTR-based device according to some embodiments of the present application. The method may be performed by the edge computing device 130 of the smart home control system. As shown in the figure, the method 400 for controlling an FTTR-based device may include the following steps:

[0096] Step 401: Obtain target intent information for the device.

[0097] Among them, the target intention information in step 401 can be the intention information for adjusting the device operating status corresponding to the user input instruction, for example, "dim the living room lights", or it can be the adjustment intention information determined by the smart home control system 110 based on the multimodal perception data of the current environment.

[0098] In an exemplary embodiment, the edge computing device 130 can obtain target intent information for the device based on multimodal perception data of the current environment and / or user input instructions. The multimodal perception data may include device status data, sensor data, device usage behavior data, and environmental monitoring data; device status data such as switch, mode, energy consumption, and commands; sensor data such as temperature, humidity, and motion; device usage behavior data such as the frequency of device use within a preset time period or the frequency of device use; and environmental monitoring data such as audio data and video data.

[0099] Step 402: Input the target intention information into the target device control model to obtain adjustment instructions for the device, wherein the target device control model is obtained by training the initial device control model through a first loss function and at least one second loss function, the first loss function is used to characterize the degree of matching between the probability values ​​of the device executing multiple candidate actions and the real action labels, and the at least one second loss function is determined based on the user preference index, and the user preference index includes at least one of the device operation energy consumption index, the device operation safety index and the user perception feedback index.

[0100] In an exemplary embodiment, the target intention information is input into the target device model, and the target device model is obtained by training the initial device control model through a first loss function and at least one second loss function, so that the target device control model can generate adjustment instructions that match the target intention information and meet user preference indicators.

[0101] Step 403: Adjust the operating state of the device through the adjustment instruction.

[0102] Continuing with the above embodiment, the edge computing device 130 adjusts the operating state of the device according to the adjustment instructions for the device output by the target device control model, for example, controlling the air-conditioning device to turn on and adjusting the air-conditioning device to 24 degrees, etc.

[0103] Through the above steps, since the initial device control model is trained by the first loss function and at least one second loss function, the trained target device control model can generate an adjustment strategy that matches the target intention information and meets the user preference indicators, thereby effectively solving the problem of deviation in the response of the smart home control system to fuzzy requests and improving the precise control capability of the smart home control system.

[0104] In some embodiments, before inputting the target intention information into the target device control model in step 402, the process further includes:

[0105] Obtain original collected data; splice the original collected data to obtain fused data; construct a fusion matrix based on the fused data, wherein the first row of the fusion matrix includes all n features, and the other rows include m features selected from all features and a zero position, m=n-1, where m is a positive integer; process the fusion matrix according to a cross-attention mechanism to obtain sample data; and send the sample data to a cloud platform so that the cloud platform constructs an initial device control model based on the sample data.

[0106] In an exemplary embodiment, the original collected data is obtained, which may include device status data such as switches and modes, sensor data such as temperature, humidity, movement, energy consumption, device usage behavior data, and environmental monitoring data such as audio data and video data. The original collected data is spliced ​​to obtain fused data. X t , as shown below:

[0107] ;

[0108] in, d is the feature dimension; t is the time window; is the first time series feature of the device status data, and its dimension is , the device state data may be normalized to obtain the first time series feature; is the second time series feature of sensor data, and its dimension is , the sensor data can be normalized to obtain the second time series feature; is the third time series feature of device usage behavior data, and its dimension is ; is the fourth time series feature of environmental monitoring data, and its dimension is Specifically, it can be the encoding of speech data or image data. Speech encoding can use open source pre-trained Transformer models, such as Wav2Vec 2.0, to extract speech features. Image encoding can use open source pre-trained CNN models, such as ResNet, VGG, EfficientNet, etc., to extract image features.

[0109] Based on fusion data X t Building a fusion matrix T Specifically, according to 、 、 、 The positions in the multimodal data matrix are sorted to generate multiple splicing sequences. For example, three splicing sequences with different orders are used as modal dimensions. The first splicing sequence, i.e., the first splicing sequence, uses all n features (in this embodiment, n=4). The second and third splicing sequences, i.e., the second splicing sequence, use randomly extracted m=n-1 features (in this embodiment, m=3). The remaining feature is set to zero. The modal combination of this feature fusion method no longer depends on a specific order, which enhances the robustness to perturbations. Different orders activate the dependency patterns between different modalities, making the feature correlation between different modalities higher during training. The resulting fusion matrix T for:

[0110] ;

[0111] According to the cross attention mechanism, the fusion matrix T The sample data is processed to obtain sample data, and the sample data is sent to the cloud platform so that the cloud platform can build an initial device control model based on the sample data.

[0112] In this way, the multimodal perception data is first fused by feature splicing to obtain fused data, and then the fused data is once again fused using the cross-attention mechanism. This can enhance the feature expression ability of the sample data, so that the initial device control model can capture the complex relationship between the correlation between multimodal data and the timing changes, which is conducive to generating device control instructions that are more in line with user intentions.

[0113] In some possible implementations, processing the fusion matrix according to a cross-attention mechanism to obtain sample data includes:

[0114] The fused data is converted into a query matrix, a key matrix and a value matrix. A preliminary score is determined based on the query matrix and the key matrix. An attention bias matrix is ​​generated based on the numbers of the n features and the position information of each feature in the fusion matrix. The attention bias matrix is ​​added to the preliminary score and then normalized to obtain the attention weight. The attention weight is multiplied by the value matrix to obtain the sample data.

[0115] In an exemplary embodiment, in the fusion encoding step, a cross-attention module with shared parameters can be used, and attention bias adjustment is performed based on permutation embedding, and a bias generated according to the modality permutation order is added to the attention score. , to enhance the connection between local modalities, such as audio and video, and suppress the invalid attention between distant modalities. The representation of sample data is as follows:

[0116] ;

[0117] in, Q represents the query matrix, K represents the key matrix, V represents the value matrix, represents the scaling factor, It is a computable attention bias matrix that can generate the attention bias matrix based on the number of n features and the position information of each feature in the fusion matrix , with the initial encoding result T For example, its attention bias matrix is:

[0118] .

[0119] The embodiment of the present application also provides a device control system based on FTTR, including a cloud platform and an edge computing device; the cloud platform is used to perform the following Figure 2 The edge computing device is used to perform the steps of the method shown in Figure 4 The steps of the method shown in .

[0120] This embodiment of the present application provides a device control method based on FTTR. By systematically preprocessing and improving multimodal data sources, namely, splicing the original collected data to obtain fused data; constructing a fusion matrix based on the fused data; and processing the fusion matrix using a cross-attention mechanism to obtain sample data, the method can improve the model's performance in user intent recognition and control command prediction tasks. The overall device control command prediction accuracy of the model increased by 7.3%, and the user satisfaction score in the control response increased by an average of 0.5 points (out of a maximum of 5 points). At the same time, the system's robustness to abnormal behavior is further enhanced, providing a data foundation and decision-making guarantee for building a smarter and more energy-efficient home intelligent system.

[0121] In an exemplary embodiment, in the above-mentioned FTTR-based device control method, a closed-loop control process is formed from the user issuing instructions to the system executing and feeding back the results. Feedback optimization samples are obtained through the closed-loop control process, and the preset neural network is trained with the feedback optimization samples to obtain the target neural network. The device control logic is generated by the target neural network, and the action execution status of the home appliances in the target area is determined. Figure 1 Taking the interactive process provided as an example, the above-mentioned FTTR-based device control method is explained. This process involves the collaboration of multiple modules, including the user layer, the Large Language Model (LLM) control module, the edge computing module, the FTTR network communication module, the device execution layer and the cloud platform layer, wherein the user layer can be deployed in the smart home control system 110; the LLM control module, the edge computing module and the device execution layer can be deployed in the edge computing device 130, and the cloud platform layer can be deployed in the cloud platform 150. Each module has its own unique functions and technical details, and the modules are closely connected through data flow and instruction transmission to ensure the efficient operation and continuous optimization of the system.

[0122] The user is the starting point and end point of the entire system, interacting with the smart home control system 110 through voice or text. The user layer is used to receive user commands and obtain the execution results and feedback corresponding to the commands. For example, a user's command to dim the living room lights will be passed to the LLM control module for processing. The uniqueness of users lies in the diversity and ambiguity of their commands. The system needs to have strong intent understanding capabilities to accurately interpret these commands. In addition, user feedback, such as "too dim", is an important source of data for system optimization.

[0123] The LLM control module is the core of the system, used to parse user commands, generate device control logic, and coordinate other modules to control the action execution status of home devices. Specifically, the LLM control module first obtains sensor data from the edge computing module and optimizes the control strategy by combining the complex task processing capabilities of the cloud platform layer. The special feature of the LLM control module lies in its multimodal intent understanding ability. It can combine language, sensor data, and camera data to generate precise device control logic, send instructions to the device execution layer through the FTTR network communication module, and receive device status feedback. In addition, the LLM control module uploads user behavior data and multimodal perception data fed back by users to the cloud platform layer, which is used for federated learning and model updates to achieve adaptive optimization of the system.

[0124] The edge computing module processes sensor data and device status locally, reducing latency and protecting user privacy. It works closely with the LLM control module to provide real-time environmental data, such as light intensity and temperature. The edge computing module is unique in its local processing capabilities, enabling it to quickly respond to LLM requests and reduce reliance on the cloud platform. It connects directly to sensors and cameras at the device execution layer to ensure real-time and accurate data.

[0125] The FTTR network communication module acts as a data transmission bridge, transferring control instructions and device status between the LLM control module and the device execution layer. The high bandwidth and low latency of the FTTR network ensure rapid transmission and execution of instructions. Its uniqueness lies in its efficiency and stability, enabling simultaneous communication with a large number of devices. The FTTR network communication module also transmits device execution layer status feedback to the LLM control module, forming a closed-loop control process.

[0126] The device execution layer, which includes various smart devices such as lights and thermostats, executes commands sent by the LLM control module via the FTTR network communication module. The device execution layer is unique in its diversity and heterogeneity; different devices may require different control protocols and interfaces. The device execution layer returns execution status in real time, such as dimming the lights, to the FTTR network communication module to ensure the system can adjust control strategies in a timely manner.

[0127] The cloud platform layer is used for model optimization; it collaborates with the LLM control module to provide powerful computing and storage capabilities. The cloud platform layer is unique in its support for federated learning, leveraging multimodal features to update and optimize the model without compromising user privacy. The optimized model is then returned to the LLM control module for improved control strategies. The cloud platform layer is used for long-term feature storage and analysis, supporting the continuous evolution of the system.

[0128] The hardware environment involved in the above-mentioned interactive system includes: (1) visual sensors: high-resolution cameras that support real-time video streaming and have night vision capabilities to ensure effective image capture under different lighting conditions; (2) audio sensors: high-sensitivity microphones that support omnidirectional sound pickup and can capture all sounds in the residence; (3) computing devices: edge computing devices, such as NVIDIA Jetson series, Raspberry Pi, etc., used to locally process video and audio data to reduce network bandwidth pressure and latency; storage devices: (4) local storage, such as NAS, used to store video frames, audio data and feature processing results; (5) various environmental sensors: such as temperature, humidity, gas, etc., used to collect the user's real environment.

[0129] The software environment involved in the above-mentioned interactive system includes: (1) operating system: embedded operating system such as Linux system, which supports multi-threading and real-time processing; (2) development framework: visual processing: OpenCV, TensorFlow, PyTorch, etc., which support CNN and Transformer models; audio processing: PaddleSpeech, Whisper, DeepSpeech, etc., which support speech classification and recognition; natural language processing: Transformers library, Hugging Face, etc., which support semantic parsing and generation; communication protocol: HTTP / HTTPS, WebSockets, MQTT, etc., which are used for data transmission and real-time communication between devices; cloud service: Alibaba Cloud, which provides object storage service (OSS), real-time messaging protocol (RTMP), message queue (MQ), model optimization and other functions;

[0130] The network environment involved in the above-mentioned interactive system includes: local area network: based on FTTR technology, ensuring low-latency data transmission between cameras and microphones and edge computing devices; wide area network: based on FTTR technology, ensuring stable data transmission between edge computing devices and the cloud; network bandwidth: at least 10 Mbps uplink bandwidth to support real-time transmission of high-definition video and audio; network delay: less than 100 ms to ensure the timeliness of real-time monitoring and alarm.

[0131] Through the configuration of the above hardware and software environment, it can run efficiently and stably in actual application scenarios, ensuring safe monitoring and timely alarm of the target area.

[0132] like Figure 5 As shown in , based on the above interactive system, its workflow may include the following steps:

[0133] The edge computing device 130 obtains the original collected data, performs fusion processing on the original collected data to obtain sample data; sends the sample data to the cloud platform so that the cloud platform builds an initial device control model based on the sample data; specifically includes steps 501 to 509:

[0134] Step 501: The user sends an instruction to the LLM control module;

[0135] Steps 502-503: The LLM control module requests and obtains multimodal perception data in response to instructions sent by the user. For example, step 5021: The LLM control module requests sensor data from the edge computing module, and step 5031: The LLM control module obtains the sensor data returned by the edge computing module. For another example, step 5022: The LLM control module requests complex task processing from the cloud platform layer, and step 5032: The LLM control module obtains the processing result returned by the cloud platform layer.

[0136] Step 504: The LLM control module performs multimodal intent understanding based on the acquired multimodal perception data and generates initial device control logic;

[0137] Step 505: Based on the initial device control logic, a control instruction is sent to the FTTR network communication module;

[0138] Step 506: The FTTR network communication module transmits the control instruction to the device execution layer;

[0139] Step 507: The device execution layer executes the control instruction and returns the execution status of the device;

[0140] Step 508: The FTTR network communication module returns the device status to the LLM control module; the LLM control module obtains the multimodal sensing data of the target area, as well as the user behavior data corresponding to the multimodal sensing data and the real action label of the device;

[0141] Step 509: The LLM control module uploads at least one of the user behavior data and the real action label of the multimodal perception data device to the cloud platform layer;

[0142] Step 510: The cloud platform layer performs federated learning and updates the preset neural network model based on the feedback optimization sample set uploaded by the LLM control module;

[0143] Specifically, the cloud platform layer trains a preset neural network model based on sample data to obtain an initial device control model, where the sample data includes multimodal perception data of the target area, and at least one of user behavior data corresponding to the multimodal perception data and a real action label of the device; the initial device control model is trained using a first loss function and at least one second loss function to obtain a target device control model, where the first loss function is used to characterize the degree of match between the probability values ​​of the device executing multiple candidate actions and the real action labels, and the at least one second loss function is determined based on a user preference index, where the user preference index includes at least one of a device operation energy consumption index, a device operation safety index, and a user perception feedback index;

[0144] Step 511: The cloud platform layer returns the optimized target device control model to the LLM control module;

[0145] Step 512: The LLM control module updates the device control logic based on the target device control model;

[0146] Step 513: The LLM control module returns the execution result to the user;

[0147] Step 514: The LLM control module obtains feedback provided by the user;

[0148] Step 515: The LLM control module adjusts the control logic according to the feedback;

[0149] Step 516: The LLM control module sends the adjusted instruction to the FTTR network communication module;

[0150] Step 517: The FTTR network communication module transmits the new instruction to the device execution layer;

[0151] Step 518: The device execution layer executes the new instruction and obtains the return device status;

[0152] Step 519: The device execution layer returns the device status to the LLM control module;

[0153] Step 520: The LLM control module returns the final result to the user.

[0154] Through the above steps, since the initial device control model is optimized by the first loss function and at least one second loss function, it can be ensured that the device can accurately perform actions that match the target intention information. At the same time, the model is adjusted through user preference indicators (such as energy consumption, safety, user feedback, etc.), so that the initial device control model can fully learn the user's preferences and needs, so that the trained target device control model can generate an adjustment strategy that matches the target intention information and meets the user preference indicators, thereby effectively solving the problem of deviation in the response of the smart home control system to fuzzy requests and improving the precise control capability of the smart home control system.

[0155] In an exemplary embodiment, Figure 6 As shown, for the user control process, the control process of the above-mentioned LLM control module includes the following steps:

[0156] Step 601: The user issues a control instruction;

[0157] Step 602: The LLM control module receives the above control instruction;

[0158] Step 603: The LLM control module parses the control instruction;

[0159] Step 604: The LLM control module collects surrounding environment information, specifically, obtains the initial device status, sensor data, device usage behavior data, and environmental monitoring data of the target area;

[0160] Step 605: The LLM control module mines the association between the control instruction and other devices. Specifically, based on the control instruction and the initial device state, it determines the device state data; based on the device state data, sensor data, device usage behavior data, and environmental monitoring data, it obtains the original collected data; it splices the original collected data to obtain fused data; it constructs a fusion matrix based on the fused data, and processes the fusion matrix using the cross-attention mechanism to obtain sample data;

[0161] Step 606: After obtaining the sample data, the LLM control module may also output user suggestions based on the sample data;

[0162] Step 607: The LLM control module obtains feedback from the user and ends the process.

[0163] In this way, the LLM control module not only analyzes the speech semantics, but also synchronously collects perception data from other available smart devices such as environmental sensor data and camera video information, and fuses these multimodal perception data to obtain sample data. Through this sample data, it can explore the potential associations between home devices, and even if the user does not explicitly mention it, it can give reasonable suggestions for the user to choose.

[0164] Optionally, after the above step 603, the following steps may be further included:

[0165] Step 608: The LLM control module parses the control instruction and sends the parsed control instruction to the device, ending the process.

[0166] In this way, when the user's control instructions are clear, the control instructions parsed by the LLM control module can be directly sent to the device to improve service responsiveness.

[0167] In some possible implementations, after inputting the target intention information into the target device control model to obtain an adjustment instruction for the device, the following steps may be further included:

[0168] By analyzing the original collected data, user motion data and device status data are obtained; based on the user motion data, the user behavior trajectory within a preset time period is determined; based on the device status data, the device's action constraints are determined; based on the above-mentioned adjustment instructions, user behavior trajectory and action constraints, device control instructions are generated and sent to the device; and user feedback information is obtained by sending query requests to the user.

[0169] In another exemplary embodiment, Figure 7 As shown, for the autonomous control process, the control process of the LLM control module includes the following steps:

[0170] Step 701: The LLM control module collects surrounding environment information and obtains original collected data;

[0171] Step 702: The LLM control module obtains user motion data and device status data by analyzing the original collected data;

[0172] Step 7031: The LLM control module performs feature aggregation and time series modeling on the user's motion data to obtain the user's behavior trajectory within a preset time period, for example, a heat map of family members' activities;

[0173] Step 7032: The LLM control module generates a device state matrix based on the device state data, such as an air conditioner energy efficiency curve and lamp life loss. Based on the device state matrix, the module determines the device's action constraints, such as a constraint satisfaction diagram.

[0174] Step 704: The LLM control module may also generate a control instruction for the device based on the adjustment instruction output by the target device control model, the user behavior trajectory, and the action constraint condition, and send the control instruction to the device;

[0175] Step 705: Obtain user feedback information by sending a query request to the user, and end the process.

[0176] In this way, the action constraints generated based on device status information (such as energy efficiency, lifespan, etc.) can ensure that the device operates under reasonable restrictions, extend the service life of the device, and improve energy efficiency; and through user behavior trajectories and action constraints, the user's needs and preferences can be accurately predicted, thereby dynamically adjusting the device's control strategy to achieve personalized and precise intelligent control of the device.

[0177] In another exemplary embodiment, Figure 8 As shown, the edge computing device 130 may also include a data collection and fusion node and an edge computing node, and the cloud platform 150 includes a cloud training collaboration node. During the data processing process of the edge computing device 130, the user provides multimodal data through multi-source sensors such as the device-side camera, microphone, temperature and humidity sensor, etc. The multimodal data may include video data, audio data, environmental data, etc. The above-mentioned FTTR-based device control method may include the following steps:

[0178] Step 801: The edge computing node obtains multimodal data collected by the user device (i.e., the original collected data mentioned above); the edge computing node splices the multimodal data to obtain fused data, constructs a fusion matrix based on the fused data, and processes the fusion matrix according to the cross-attention mechanism to obtain sample data;

[0179] Specifically, the edge computing node first aligns the timestamps and spatial features of the multi-source sensor data in the multimodal data to generate a composite feature vector. During the multimodal data fusion phase, the edge computing node uses convolutional neural networks and recurrent neural networks to extract visual and speech features, respectively, and generates fused data through a feature fusion layer (attention mechanism). Sample data is obtained based on the composite feature vector and fused data. Optionally, a user feedback mechanism, such as click and voice feedback, can be set up during this data fusion phase to optimize the feature extraction and fusion process.

[0180] Step 802: The edge computing node encrypts the sample data and uploads the encrypted sample data to the cloud training coordination node;

[0181] Specifically, in the privacy protection phase, edge computing nodes can use local differential privacy technology to add noise during the feature extraction phase to ensure that individual data is irreversible. For sensitive features, such as user behavior patterns, homomorphic encryption technologies such as Paillier encryption are used to achieve "data availability but invisibility", ensuring that data can still be calculated in an encrypted state. Finally, edge nodes use lightweight quantization models such as INT8 quantization and NEON instruction set acceleration technology to control end-to-end latency within an imperceptible range and upload encrypted sample data to the cloud training collaboration node.

[0182] Step 803: The edge computing node obtains the target device control model sent by the cloud training collaboration node;

[0183] Specifically, in the model training and optimization steps of the cloud training collaborative node, after the cloud training collaborative node receives the encrypted sample data uploaded by the edge computing node, it can use the distributed training framework to perform model training; during the training process, the Secure Aggregation protocol is used to encrypt and transmit the gradient, ensuring that the edge nodes participating in the training only expose the sum of the model parameters to resist man-in-the-middle attacks; in the model quantization stage, the cloud training collaborative node compresses the trained global model into a lightweight model through quantization technology, and converts the models exported from different edge frameworks into a federated learning standard format, such as ONNX, through the model adaptation layer to eliminate platform dependencies; in the knowledge distillation stage, the cloud training collaborative node uses the Teacher-Student model to convert the complex global model into a lightweight edge model, and sends the optimized model to the edge computing node;

[0184] Step 804: The edge computing node obtains the fused data sent by the data collection and fusion node;

[0185] Specifically, in the multimodal feature generation step of the data collection and fusion node, the data collection and fusion node first defines a unified data specification, covering multi-dimensional data sources such as device status, environmental perception (for example, temperature, humidity, and lighting), and user interaction. By collecting data from all scenarios, the diversity and comprehensiveness of the data are ensured; in the feature extraction stage, the data collection and fusion node uses MobileNet-SSD to extract scene semantic features, combines the optical flow method to capture dynamic changes, and establishes semantic associations between multimodal features through the attention mechanism; in the multimodal feature fusion stage, the data collection and fusion node uses the Transformer model to fuse visual features, temporal features, and cross-modal associations to generate high-quality multimodal features, and provides the fused data to the edge computing node;

[0186] Step 805: After completing data processing and model optimization, the edge computing node returns the real-time processing results to the user; the user further optimizes the system performance through interactive feedback, such as clicks and voice feedback, forming a closed loop.

[0187] In this way, the system response speed can be improved, bandwidth allocation can be improved, and the security of device-side data can be ensured.

[0188] Figure 9A schematic diagram of the hardware structure of an electronic device that implements the embodiments of the present application is shown. Referring to this diagram, at the hardware level, electronic device 900 includes a processor 910, and optionally, an internal bus 920, a network interface 930, and a memory. The memory may include internal memory 941, such as high-speed random-access memory (RAM), and may also include non-volatile memory 942, such as at least one disk storage device. Of course, electronic device 900 may also include hardware required for other services.

[0189] The processor 910, network interface 930, and memory can be interconnected via an internal bus 920. This internal bus 920 can be an Advanced Microcontroller Bus Architecture (AMDBA) bus, a Wishbone bus, an Open Core Protocol (OCP) bus, an Avalon bus, or the like. Such buses can be categorized as address buses, data buses, and control buses. For ease of illustration, this figure uses only one bidirectional arrow, but this does not imply that there is only one bus or only one type of bus.

[0190] The memory stores programs. Specifically, the programs may include program codes, which include computer operating instructions. The memory may include internal memory 941 and non-volatile memory 942, and provides instructions and data to the processor 910.

[0191] The processor 910 reads the corresponding computer program from the non-volatile memory 942 into the memory and then runs it, forming a device for locating the target user at the logical level. The processor 910 executes the program stored in the memory and specifically performs the following: Figure 2 or Figure 4 The methods disclosed in the illustrated embodiments implement the functions and beneficial effects of the various methods described in the foregoing method embodiments, which will not be described in detail here.

[0192] The above application Figure 2 or Figure 4The methods disclosed in the illustrated embodiments can be applied to or implemented by processor 910. Processor 910 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be performed by hardware integrated logic circuits or software instructions in processor 910. The processor 910 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The methods, steps, and logic block diagrams disclosed in the embodiments of this application can be implemented or executed. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above method.

[0193] The computer device can also execute the methods described in the above method embodiments and realize the functions and beneficial effects of the methods described in the above method embodiments, which will not be repeated here.

[0194] Of course, in addition to software implementation, the electronic device 900 of the present application does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0195] The embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores one or more programs, which, when executed by an electronic device including a plurality of application programs, enables the electronic device to execute Figure 2 or Figure 4 The methods disclosed in the illustrated embodiments implement the functions and beneficial effects of the various methods described in the foregoing method embodiments, which will not be described in detail here.

[0196] The computer-readable storage medium includes a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0197] Furthermore, an embodiment of the present application provides a computer program product, comprising a computer program stored on a non-transitory computer-readable storage medium, wherein the computer program comprises program instructions. When the program instructions are executed by a computer, the following process is implemented: Figure 2 or Figure 4 The methods disclosed in the illustrated embodiments implement the functions and beneficial effects of the various methods described in the foregoing method embodiments, which will not be described in detail here.

[0198] The embodiments of the present application can be applied to various electronic device collaboration or interconnection scenarios, including: collaboration and interconnection between mobile phones and laptops / tablets; collaboration and interconnection between mobile terminals and smart TVs / displays; collaboration and interconnection between mobile phones or tablets and in-car entertainment systems; collaboration and interconnection between mobile terminals and smart conference systems, etc., thereby meeting the diverse needs of users in scenarios such as smart homes, smart offices, and smart travel.

[0199] In short, the above description is only a preferred embodiment of the present application and does not limit the scope of protection of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

[0200] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0201] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0202] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0203] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the method embodiments.

Claims

1. A device control method based on FTTR, characterized in that: include: Obtaining an initial device control model, wherein the initial device control model is used to adjust the operating state of the device according to the target intention information, and the device is provided with an edge computing module; The initial device control model is trained using a first loss function and at least one second loss function to obtain a target device control model, wherein the first loss function is used to characterize the degree of match between probability values ​​of the device executing multiple candidate actions and real action labels, and the at least one second loss function is determined based on a user preference index, wherein the user preference index includes at least one of a device operation energy consumption index, a device operation safety index, and a user perception feedback index; sending the target device control model to the device; Among them, the second loss function includes a fifth loss function corresponding to the user perception feedback index; the fifth loss function includes a third loss term, a fourth loss term and a fifth loss term, the third loss term is used to characterize the degree of difference between the real user preference probability distribution and the predicted user preference probability distribution, the real user preference probability is determined according to the adjustment intention information, and the adjustment intention information is the user feedback information after adjusting the operating state of the device based on the target device control model, the fourth loss term is used to characterize the degree of difference between multiple candidate actions and user control behaviors, and the fifth loss term is used to characterize the degree of consistency between the comfort prediction results of the first scene and the comfort prediction results of the second scene, and the difference between the environmental parameters of the first scene and the second scene is less than the preset parameter threshold.

2. The method according to claim 1, characterized in that The second loss function includes a third loss function corresponding to the energy consumption index of the device operation; the third loss function includes a first loss term and a second loss term, the first loss term is used to characterize the degree of difference between the predicted power and the actual power when the device performs the multiple candidate actions, and the second loss term is used to characterize the degree of difference between the predicted average power and the actual average power of the device based on the comfort estimate compensation within a preset time period, the comfort estimate is determined based on historical energy consumption data and the user comfort score, and the user comfort score is determined based on the multimodal perception data in the target intention information.

3. The method according to claim 1, characterized in that The second loss function includes a fourth loss function corresponding to the device operation safety index; the fourth loss function is used to characterize the safety level of the device performing a target action among multiple candidate actions.

4. The method according to any one of claims 1 to 3, characterized in that The training of the initial device control model by using the first loss function and the at least one second loss function to obtain the target device control model includes: Determining, according to each user preference index and the priority of the first loss function, a preset number of target loss functions with higher priorities among a plurality of loss functions, the plurality of loss functions including the first loss function and the second loss function; Determining the weights of the multiple loss functions according to a preset condition, wherein the preset condition is that the weight of each target loss function is set within a preset weight range corresponding to each target loss function; The initial device control model is trained according to the multiple loss functions and the weights corresponding to the respective loss functions to obtain a target device control model.

5. The method according to claim 1, wherein The obtaining of the initial device control model includes: Acquire sample data, the sample data including multimodal perception data of the target area, and at least one of user behavior data corresponding to the multimodal perception data and a real action label of the device; The preset neural network model is trained according to the sample data to obtain the initial equipment control model.

6. The method according to claim 5, characterized in that The obtaining of sample data includes: Receive sample data sent by the device, wherein the sample data is obtained by the device splicing original collected data to obtain fused data, constructing a fusion matrix based on the fused data, and processing the fusion matrix according to a cross-attention mechanism.

7. The method according to claim 5, characterized in that The preset neural network includes a plurality of sub-networks corresponding to target functional tasks one by one; the training of the preset neural network model based on the sample data to obtain the initial device control model includes: Iteratively training the preset neural network using sample data, and obtaining the network parameters of each sub-network and the exponential moving average of the loss function gradient corresponding to each sub-network during each iterative training process, wherein the exponential moving average is used to characterize the sensitivity of the network parameters of the sub-network to the loss function; Determine the activation mask of each sub-network based on the network parameters and exponential moving average of each sub-network; A target subnetwork whose activation mask meets a preset threshold is selected from the subnetworks, the target subnetwork is trained using the sample data, and the network parameters of the subnetworks and the exponential moving average of the loss function gradient corresponding to the subnetworks are updated until the preset convergence conditions are met, and the preset neural network after convergence is determined as the initial device control model.

8. The method according to claim 7, characterized in that The step of determining the activation mask of each sub-network according to the network parameters and the exponential moving average of each sub-network includes: Multiply the network parameters of each sub-network and the exponential moving average gradient value element by element to obtain the target product value; Taking the average of the absolute values ​​of the target product values ​​to obtain an estimated parameter influence value of each sub-network, wherein the estimated parameter influence value is used to indicate the degree of influence of the sub-network on the update of network parameters during the training process; Normalizing the estimated parameter influence to obtain an activation probability; The activation probability of each sub-network is sampled to obtain an activation mask of each sub-network.

9. A device control method based on FTTR, characterized in that: include: Get target intent information for the device; Inputting the target intention information into a target device control model to obtain an adjustment instruction for the device, wherein the target device control model is obtained by training an initial device control model using a first loss function and at least one second loss function, the first loss function being used to characterize the degree of match between probability values ​​of the device executing multiple candidate actions and real action labels, and the at least one second loss function being determined based on a user preference index, the user preference index including at least one of a device operation energy consumption index, a device operation safety index, and a user perception feedback index; Adjusting the operating state of the device through the adjustment instruction; Among them, the second loss function includes a fifth loss function corresponding to the user perception feedback index; the fifth loss function includes a third loss term, a fourth loss term and a fifth loss term, the third loss term is used to characterize the degree of difference between the real user preference probability distribution and the predicted user preference probability distribution, the real user preference probability is determined according to the adjustment intention information, and the adjustment intention information is the user feedback information after adjusting the operating state of the device based on the target device control model, the fourth loss term is used to characterize the degree of difference between multiple candidate actions and user control behaviors, and the fifth loss term is used to characterize the degree of consistency between the comfort prediction results of the first scene and the comfort prediction results of the second scene, and the difference between the environmental parameters of the first scene and the second scene is less than the preset parameter threshold.

10. The method according to claim 9, characterized in that Before inputting the target intention information into the target device control model, the method further includes: Obtain original collected data; splicing the original collected data to obtain fused data; Constructing a fusion matrix according to the fusion data, wherein the first row of the fusion matrix includes all n features, and the other rows include m features selected from all features and a zero position, where m=n-1, and m is a positive integer; Processing the fusion matrix according to the cross attention mechanism to obtain sample data; The sample data is sent to a cloud platform so that the cloud platform constructs an initial device control model according to the sample data.

11. The method according to claim 10, characterized in that The fusion matrix is ​​processed according to the cross attention mechanism to obtain sample data, including: converting the fused data into a query matrix, a key matrix, and a value matrix, and determining a preliminary score based on the query matrix and the key matrix; Generate an attention bias matrix according to the numbers of the n features and the position information of each feature in the fusion matrix; Adding the attention bias matrix to the preliminary score and performing normalization to obtain an attention weight; Multiply the attention weight by the value matrix to obtain sample data.

12. A device control system based on FTTR, characterized in that: It includes a cloud platform and an edge computing device; the cloud platform is used to execute the steps of the method according to any one of claims 1 to 8, and the edge computing device is used to execute the steps of the method according to any one of claims 9 to 11.

13. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the method according to any one of claims 1 to 11 are implemented.

14. A computer program product, characterized in that The computer program product comprises a computer program stored on a non-transitory computer-readable storage medium, wherein the computer program comprises program instructions, which, when executed by a computer, cause the computer to perform the steps of the method according to any one of claims 1 to 11.

Images