Equipment control method and device, equipment and computer storage medium

Abstract

The invention relates to the technical field of equipment control, and provides an equipment control method and device, equipment and a computer storage medium. The method comprises the following steps: acquiring equipment state information, collected by a multi-mode sensor, of the electronic equipment; identifying the equipment state information to obtain a plurality of atomic states; according to the multiple atomic states, a pre-constructed state map is inquired, multiple state sequences are obtained, nodes of the state map represent the atomic states, edges of the state map represent the incidence relation between the states, and weights of the edges represent the historical execution effect of a task corresponding to the conversion from the precursor state to the subsequent state; determining a target state sequence from the plurality of state sequences; the target state sequence is analyzed into at least one equipment control task, and the equipment control task is used for indicating a control action executed by the electronic equipment so as to achieve a state evolution target defined by the target state sequence; and controlling the electronic equipment according to the equipment control task.

Description

Technical Field

[0001] This invention belongs to the field of equipment control technology, and particularly relates to an equipment control method, device, equipment, and computer storage medium. Background Technology

[0002] In today's context of rapid technological advancements, intelligent control of electronic equipment has become a core requirement in industrial automation and precision manufacturing. From precision instruments on complex production lines to distributed field monitoring terminals, autonomous, precise, and environmentally adaptable control of equipment is key to improving system reliability, efficiency, and intelligence.

[0003] Currently, mainstream equipment control technologies largely rely on pre-set programming logic, fixed rule bases, or control models trained under specific operating conditions. These methods can achieve efficient control in known and stable operating scenarios. However, in practical applications, electronic devices often face complex challenges such as variable tasks, dynamic environments, and non-standard states. Traditional methods struggle to enable control systems to autonomously understand new tasks, respond to unforeseen state changes, or continuously evolve from operational experience without human intervention. This results in limited adaptability of existing systems, and their intelligence and autonomy often fall short of requirements in open, uncertain "outdoor" or non-standard industrial scenarios.

[0004] Therefore, it is currently difficult for electronic devices to achieve intelligent autonomous control in complex and ever-changing environments. Summary of the Invention

[0005] This invention provides a device control method, apparatus, device, and computer storage medium that enable intelligent autonomous control.

[0006] In a first aspect, embodiments of the present invention provide a device control method, the method comprising: Acquire device status information of electronic devices collected by multimodal sensors; The device status information is identified to obtain multiple atomic states; Based on multiple atomic states, a pre-constructed state graph is queried to obtain multiple state sequences. In the state graph, nodes represent atomic states, edges represent the relationships between states, and the weight of the edges represents the historical execution effect of the task corresponding to the transition from the predecessor state to the successor state. The target state sequence is determined by evaluating multiple state sequences using a reinforcement learning strategy. The target state sequence is parsed into at least one device control task, wherein the device control task is used to instruct the electronic device to perform control actions to achieve the state evolution target defined by the target state sequence; Control the electronic equipment according to the equipment control task.

[0007] In a second aspect, embodiments of the present invention provide a device control apparatus, the apparatus comprising: The acquisition module is used to acquire device status information of electronic devices collected by multimodal sensors; The identification module is used to identify device status information and obtain multiple atomic states; The query module is used to query a pre-built state graph based on multiple atomic states to obtain multiple state sequences. The nodes of the state graph represent atomic states, the edges of the state graph represent the relationships between states, and the weight of the edge represents the historical execution effect of the task corresponding to the transition from the predecessor state to the successor state. The evaluation module is used to evaluate multiple state sequences using reinforcement learning strategies and determine the target state sequence from the multiple state sequences. The parsing module is used to parse the target state sequence into at least one device control task, wherein the device control task is used to instruct the control actions to be performed by the electronic device to achieve the state evolution target defined by the target state sequence; The control module is used to control electronic equipment according to the equipment control tasks.

[0008] A third aspect of the present invention provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described method.

[0009] In a fourth aspect, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method.

[0010] Fifthly, embodiments of the present invention provide a computer program product, including a computer program, which, when executed by a processor, implements any of the device control methods described in the above embodiments.

[0011] The device control method, apparatus, device, and computer storage medium of this invention acquire device state information of electronic devices collected by multimodal sensors; identify the device state information to obtain multiple atomic states; and achieve a unified and structured representation of the operating conditions of complex devices, transforming the originally heterogeneous and continuous low-level signals into high-level, reasonable symbolic information, laying the foundation for subsequent knowledge-based processing. Multiple state sequences are obtained by querying a pre-constructed state graph based on multiple atomic states. In the state graph, nodes represent atomic states, edges represent the relationships between states, and the weights of the edges represent the historical execution effects of the tasks corresponding to the transition from the predecessor state to the successor state. This is equivalent to encoding historical experience into a queryable knowledge network and performing target reasoning based on the current context. It can proactively plan the evolution path of the device's target state using prior knowledge, realizing a shift from passive response to proactive planning. Multiple state sequences are evaluated using a reinforcement learning strategy to determine a target state sequence. This target state sequence is then parsed into at least one device control task, which instructs the electronic device to perform control actions to achieve the state evolution objective defined by the target state sequence. The abstract target state sequence is then parsed into specific device control tasks and executed, thereby controlling the electronic device according to these tasks. This reduces reliance on external preset instructions or manual programming. By incorporating historical experience into the knowledge base, the autonomous response capability and scientific decision-making of the electronic device in dynamic operating conditions are enhanced, achieving intelligent control of the electronic device. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments of the present invention will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a flowchart illustrating a device control method provided in an embodiment of the present invention; Figure 2 This is a flowchart illustrating another device control method provided in an embodiment of the present invention; Figure 3 This is a flowchart illustrating a device control system provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a device control apparatus provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0014] The features and exemplary embodiments of various aspects of the present invention will now be described in detail. To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely intended to explain the present invention and not to limit the present invention. For those skilled in the art, the present invention can be practiced without some of these specific details. The following description of the embodiments is merely to provide a better understanding of the present invention by illustrating examples of the invention.

[0015] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0016] In complex and ever-changing industrial environments, a large number of devices need to operate continuously under unstructured and non-standardized "field" conditions. These environments are often difficult to model accurately in advance, and the operating conditions change dynamically, making it difficult for traditional control systems based on fixed rules or pre-programmed systems to adapt.

[0017] This invention aims to address the aforementioned challenges, enabling AI agents to autonomously discover skills and control equipment in such scenarios. The agent perceives equipment status in real time through multimodal sensors and, based on a built-in knowledge graph and reinforcement learning mechanism, autonomously generates control tasks adapted to the current situation. It can learn effective operating strategies not only from historical data and real-time interactions but also, when encountering unfamiliar working conditions or tasks, discover new and safe coping methods through segmented evaluation and strategy exploration, transforming this experience into reusable knowledge. This process allows the agent to gradually accumulate skills, optimize decisions, and ultimately achieve reliable, adaptive, and continuously evolving autonomous control of equipment in "wilderness" industrial environments lacking comprehensive prior models and frequent human intervention.

[0018] To address the problems of the prior art, embodiments of the present invention provide a device control method, apparatus, device, and computer storage medium.

[0019] The device control method provided in the embodiments of the present invention will be introduced first below.

[0020] Figure 1 A flowchart illustrating a device control method according to an embodiment of the present invention is shown. Figure 1 As shown, the method may include the following steps: Step 110: Obtain device status information of the electronic device collected by the multimodal sensor; Step 120: Identify the device status information to obtain multiple atomic states; Step 130: Query the pre-constructed state graph based on multiple atomic states to obtain multiple state sequences. The nodes of the state graph represent atomic states, the edges of the state graph represent the relationships between states, and the weight of the edge represents the historical execution effect of the task corresponding to the transition from the predecessor state to the successor state. Step 140: Evaluate multiple state sequences using a reinforcement learning strategy to determine the target state sequence from the multiple state sequences; Step 150: The target state sequence is parsed into at least one device control task, wherein the device control task is used to instruct the electronic device to perform control actions to achieve the state evolution target defined by the target state sequence. Step 160: Control the electronic equipment according to the equipment control task.

[0021] Equipment status information refers to a collection of multi-dimensional physical signals synchronously collected by various types of sensors deployed on electronic devices, such as vision sensors, acoustic sensors, temperature sensors, and mechanical sensors, comprehensively reflecting the real-time operating conditions of the equipment. An atomic state is the smallest unit state description identified and extracted from the aforementioned multi-dimensional status information, used to describe a specific dimension of the equipment, such as equipment temperature value, equipment vibration amplitude, or equipment load percentage. A state graph is a pre-built knowledge representation model with a directed graph structure. Each node corresponds to a specific atomic state, and the edges connecting two nodes represent potential operation paths from the source node state to the target node state. The weight value associated with each edge quantifies the average execution effect of the historical tasks corresponding to the state transition operation, such as average time consumption, average energy consumption, or average success rate.

[0022] In step 110, raw device status information of the electronic device is acquired in real time from the multimodal sensor network. This information is the data basis for all subsequent intelligent decisions.

[0023] In step 120, the acquired heterogeneous state information is processed and identified, decomposed and mapped into a series of discrete, semantically clear atomic states. For example, two independent atomic states, "the real-time temperature of the motor winding is 85 degrees Celsius" and "the current speed of the coolant circulation pump is 1200 revolutions per minute," are identified from the sensor data stream.

[0024] Step 130 involves querying and reasoning within a pre-constructed state graph based on the identified multiple atomic states. Using the currently identified set of atomic states as input to the graph, and leveraging the transition relationships and historical effect weights recorded in the graph, multiple possible future state evolution sequences are dynamically generated through graph traversal or probabilistic reasoning algorithms. A state sequence describes the complete evolution path of the device from its current high-temperature state, through a cooling operation, to finally entering a safe temperature range.

[0025] In step 140, a pre-trained reinforcement learning policy model is invoked to evaluate each candidate state sequence generated in step 130. This reinforcement learning policy incorporates a reward function that comprehensively considers multiple optimization objectives such as task completion, estimated time, and resource consumption, calculating a comprehensive reward value for each state sequence. The state sequence with the highest reward value is selected as the target state sequence, representing the optimal device execution path under the current environment.

[0026] Step 150 is responsible for instantiating the abstract target state sequence into specific control instructions. It parses the operational intent implied by each state transition in the target state sequence and transforms it into one or more device control tasks that can be directly recognized and executed by the electronic device controller. For example, if the target sequence requires the device to transition from a "high temperature state" to a "normal temperature state," the parsed and generated device control tasks could be a set of specific instructions, such as "increase the cooling fan power to 70%" and "start the secondary cooling cycle."

[0027] In step 160, the generated device control task is sent to the underlying actuator of the electronic device to drive it to operate according to a predetermined instruction sequence, thereby achieving precise control of the electronic device.

[0028] By employing multimodal fusion sensing and atomic state recognition, a high-dimensional, fine-grained unified representation of the operational state of complex electronic devices is achieved, providing reliable input for accurate decision-making. By introducing and querying pre-built state graphs, historical experience is structurated and organized, enabling efficient reasoning using prior knowledge to quickly generate candidate target solutions that conform to physical laws and operational conventions, significantly reducing the ineffective search space. A reinforcement learning strategy is used to globally optimize and evaluate candidate state sequences, ensuring the selected target achieves optimal performance across multiple metrics. A distinction is made between the target state sequence and the device control task; the target state sequence represents the desired state, while the device control task represents the actions to be performed. This reduces reliance on manual programming and real-time intervention, significantly improving the autonomous response capability and operational efficiency of electronic devices in the face of changes in routine operating conditions.

[0029] In one embodiment, step 120 may specifically include the following steps: The equipment status information is processed in a structured manner to generate structured equipment status information; The state information of structured equipment is identified and decomposed into multiple atomic states.

[0030] Raw device status information typically originates from sensors of different types and interfaces, with varying formats, frequencies, and physical meanings. Examples include continuous voltage signals from temperature sensors, spectral waveforms from vibration sensors, RGB image frames captured by cameras, and acoustic time-domain signals recorded by microphones. The goal of structured processing is to transform these heterogeneous, non-standardized raw data streams into a unified, regular, and machine-readable data organization format. This process may specifically include data synchronization, timestamp alignment, format conversion, dimension normalization, and organizing data from different modalities according to preset fields or channels, ultimately generating structured device status information. For example, a structured information unit might contain fields such as "Timestamp: T1," "Temperature Channel: 85.3 degrees Celsius," "Vibration Channel: Peak frequency 125Hz, amplitude 0.15mm," "Visual Channel: Device panel indicator A is green, indicator B is red," and "Acoustic Channel: Average sound pressure level 65dB, no abnormal spikes."

[0031] After obtaining the structured equipment status information, the next step is to perform identification and decomposition steps to extract multiple atomic states. This step utilizes pre-trained models or predefined rules to perform semantic parsing and status determination on each field or combination in the structured information. It scans the structured data and identifies the smallest granularity of state description that can independently characterize the operating status of a specific dimension of the equipment. Each identified atomic state should be a clear, unambiguous assertion describing a single attribute. For example, from the above structured information, the following independent atomic states can be identified and decomposed: "The current temperature of the equipment is 85.3 degrees Celsius," "The current vibration frequency of the equipment is 125 Hz," "Indicator light A on the equipment panel is lit and green," "Indicator light B on the equipment panel is lit and red," and "The noise level of the equipment is 65 decibels." These atomic states together constitute a discretized and symbolic set of descriptions of the equipment's current complex operating condition.

[0032] Structured processing solves the challenge of fusing data from multi-source heterogeneous sensors, providing clean and consistent input for subsequent unified analysis and forming the foundation for building a reliable perception layer. It elevates raw sensing data into atomic states with clear semantics, achieving a transformation from low-level signals to high-level semantics. This allows subsequent reasoning and task generation based on state graphs to be performed at a conceptually clear symbolic level, improving the interpretability of cognition and decision-making. Decomposing data into atomic states enables flexible combination and utilization of state information from different dimensions, enhancing the fine-grained understanding of device states and supporting the generation of more precise and adaptive control strategies.

[0033] In one embodiment, step 140 may specifically include the following steps: The reward value for each sequence of states is obtained by evaluating multiple state sequences using a reinforcement learning strategy. The state sequence with the highest reward value is determined as the target state sequence.

[0034] Reinforcement learning strategies comprehensively evaluate each input state sequence based on its embedded reward function. This reward function is designed to quantify the quality of a state sequence, and its calculation comprehensively considers multiple preset technical indicators related to the state evolution process described by the sequence, such as expected task completion rate, estimated action sequence time, and predicted resource consumption efficiency.

[0035] After evaluation, a quantified reward value is output for each candidate state sequence, which directly reflects the expected utility of executing the sequence under the policy. Next, after the reward values ​​for all candidate state sequences have been calculated, a comparison and selection operation is performed, comparing all reward values ​​and identifying the one with the highest value.

[0036] By evaluating multiple state sequences using a reinforcement learning strategy, a reward value is obtained for each of the multiple state sequences. The state sequence corresponding to the highest reward value is determined as the target state sequence, ensuring that the final selected target is the solution with the highest expected return among all possible paths under the optimization objective defined by the reinforcement learning strategy.

[0037] In one embodiment, after step 140, the following steps may be included: If the reward value of the target state sequence is higher than the preset threshold and the abstract task information corresponding to the target state sequence is not stored in the task library, the target state sequence is abstracted to generate abstract task information. Store abstract task information in the task library; Based on the device feedback information, the weights of the corresponding edges in the state graph are updated. The device feedback information is generated by the electronic device after it performs a device control task.

[0038] Once the target state sequence is determined, a self-directed knowledge update process can be executed. This process is triggered by the simultaneous fulfillment of two logical checks: First, the calculated reward value of the target state sequence must be higher than a pre-set reward threshold, which is used to filter out experiences with sufficiently good results that are worth recording; second, it needs to be confirmed that the experience represented by the target state sequence is not included in the existing knowledge base, specifically based on the absence of its corresponding abstract task information in the task library. If both conditions are met, the process of new task discovery and knowledge accumulation will begin.

[0039] This high-value target state sequence is abstracted. This process aims to strip away the details in the sequence that are overly tied to specific devices, times, or specific parameter values, and extract its general state transition logic and target pattern, thereby generating a reusable abstract task information. For example, "reducing the temperature of model A motor from 85 degrees to 70 degrees" can be abstracted into "performing a gradient cooling operation on high-temperature equipment".

[0040] The generated abstract task information is stored as a new knowledge entry in a task repository dedicated to managing experience knowledge, thus archiving the new skill. To simultaneously optimize the state graph used to generate candidate sequences, the edge weights in the graph are updated based on the actual effect feedback information generated after the device executes the control task parsed from the target state sequence. Specifically, all edges in the graph involved in realizing this state transition are located, and then, based on the actual feedback from this execution, such as the actual task completion time, energy consumption, or stability indicators, the original historical execution effect weights of those edges are dynamically adjusted.

[0041] This allows the state graph to continuously absorb new practical experience, and its internal weight values ​​can more accurately reflect the real utility of different operation paths in the current environment, thereby providing a more reliable basis for future decision-making and ultimately realizing the continuous evolution of the agent's policy capabilities.

[0042] like Figure 2 As shown, in one embodiment, step 150 may specifically include the following steps: Step 210: Generate multiple dynamic strategies based on the device control task, wherein each dynamic strategy contains an action sequence consisting of multiple consecutive actions; Step 220: Group the multiple dynamic strategies according to the similarity of their action trajectories to obtain multiple strategy groups; Step 230: Within each strategy group, determine the benefit value of the dynamic strategy based on the simulation information of the dynamic strategy; the simulation information includes: simulated device feedback information, action time, resource consumption, and task completion rate; Step 240: Within each strategy group, the dynamic strategy with the highest return value is determined as the target dynamic strategy. Step 250: Control the electronic device based on the target dynamic strategy.

[0043] A dynamic strategy refers to a complete operational plan designed to accomplish equipment control tasks. Its core is a sequence of actions composed of multiple control commands arranged in a specific time order, such as a coherent operational chain including three steps: "increasing power, monitoring temperature, and adjusting cooling." The similarity of action trajectories is used to measure the degree of closeness in the execution logic of different dynamic strategies. This judgment can be based on the order of the action types included in the strategy, the target equipment component being operated on, or a preset state transition path. Simulation information refers to the set of process and result data expected to be generated by executing a dynamic strategy in a virtual or theoretical model before executing actual control. The benefit value is a comprehensive quantitative indicator used to evaluate the overall merits of a dynamic strategy in a simulation environment; its calculation integrates multiple preset performance dimensions.

[0044] Based on the analyzed equipment control task, multiple alternative dynamic strategies are generated. This step aims to explore different technical paths to achieve the same task objective. For example, for a control task of "reducing the temperature inside the reactor," an aggressive strategy primarily focused on "increasing the cooling water flow rate" and a moderate strategy primarily focused on "slightly reducing the heating power and supplementing it with fan cooling" can be generated simultaneously. Each strategy contains a series of continuous and logically consistent specific operation instructions. Next, all generated dynamic strategies are categorized and grouped according to the similarity of their action trajectories, forming multiple strategy groups. This operation is a structured organization of the strategy space; for example, all strategies that mainly rely on adjusting cooling are grouped into the "cooling adjustment group," and all strategies that mainly achieve temperature control by adjusting heating power are grouped into the "power adjustment group."

[0045] After grouping, within each strategy group, simulation information is used to evaluate each dynamic strategy to determine its benefit value. Each strategy is executed separately in a simulation environment, and the device feedback information generated by the simulation is collected, such as simulated temperature change curves and pressure values; simultaneously, the simulation action time, estimated resource consumption such as electrical energy, and the simulation task completion rate are recorded. Based on this simulation information, a quantified benefit value is assigned to each strategy using a predefined benefit calculation model.

[0046] Within each group, the dynamic strategy with the highest return value is selected as the target dynamic strategy for that group, ensuring that the optimal representative solution under the current simulation conditions is selected from each technical path category. Finally, based on these target dynamic strategies selected for each group, the electronic equipment is actually controlled.

[0047] By generating multiple dynamic strategies for a single control task, the limitations of traditional single-path planning are overcome, significantly expanding the search space for solutions and enhancing strategy flexibility in complex or uncertain operating conditions. A strategy grouping mechanism based on motion trajectory similarity is introduced, effectively classifying and organizing massive candidate strategies, avoiding blind comparisons during the evaluation process, making the evaluation work more orderly and efficient, and also helping to identify and understand different solution approaches. Within each group, benefit evaluation and competitive selection are conducted based on multi-dimensional simulation information, ensuring that the final selected target dynamic strategy is not only a globally optimal solution but also a locally optimal solution within its respective technical path category, thus balancing strategy diversity and superiority. By decomposing complex equipment control problems into manageable and evaluable steps, the reliability and intelligence level of automated decision-making are significantly improved.

[0048] In one embodiment, step 210 may specifically include the following steps: The device control task is matched with nodes in a pre-built causal relationship graph, where nodes represent device states, edges represent actions, and edge weights represent device feedback. The pre-built causal relationship graph is obtained by training a graph neural network on historical state information, historical action sequences, and device feedback signals. Path search is performed in the causal relationship graph to obtain multiple action paths; Each action path is treated as a dynamic policy to obtain multiple dynamic policies.

[0049] A causal graph is a pre-built knowledge structure that encodes the relationships between states, actions, and results in a device's operating environment in the form of a graph. Specifically, each node in the graph represents a specific device state, such as "motor running at high speed" or "water tank level below the warning line." A directed edge connecting two nodes represents an action that can cause a state transition, such as "closing the valve" or "starting the pump." The weight associated with each edge quantifies the typical device feedback obtained after performing this action in historical experience, such as the increase in efficiency or the increase in energy consumption. This causal graph is not manually created but automatically learned by training a graph neural network on a large amount of historical data. The training data includes snapshots of historically recorded state information, sequences of actions performed in those states, and observed device feedback signals after execution. The graph neural network learns and summarizes stable mapping relationships between states, actions, and feedback from this data, ultimately constructing a graph model that reflects the inherent causal dynamics.

[0050] The device control task to be executed is semantically or target-wise matched with nodes in a pre-built causal relationship graph. For example, if the device control task is "to reduce the reactor temperature to a safe range", the graph will look for a node representing "reactor high temperature" as the starting search point and a node representing "reactor normal temperature" as one of the target points.

[0051] Based on the successfully matched nodes, path search is performed in the causal relationship graph. Using a graph search algorithm, starting from the initial state node, subsequent state nodes reachable through different action edges are traversed to find all feasible paths to the target state node. Each complete path from the starting point to the target point consists of a series of alternating state nodes and action edges, representing a complete process of continuous state transitions triggered by consecutive actions. For example, a path can be found as follows: from the "high temperature" state node, through the "open cooling water valve" action edge, to the "temperature begins to drop" state node, then through the "increase cooling pump power" action edge, finally reaching the "normal temperature" state node.

[0052] Each action path obtained from the search is directly mapped to a dynamic policy. Specifically, the action instructions corresponding to the sequentially connected action edges in the path are extracted and combined to form the action sequence of the policy; while the sequence of state nodes traversed by the path defines the state evolution trajectory that the policy expects to guide the device through. In this way, multiple different action paths are transformed into multiple alternative dynamic policies.

[0053] This approach, based on causal graph generation of dynamic strategies, utilizes explicit causal models learned from historical data for reasoning. This ensures that the strategy generation process is grounded in interpretable causal relationships, improving the rationality and reliability of the strategies. Path searching within the causal graph essentially transforms the strategy generation problem into a pathfinding problem within a structured knowledge space. It can enumerate all potential causal chains leading from the initial state to the target state, ensuring the diversity and coverage of strategies. Since the edge weights in the graph carry historical feedback information, the historical utility of actions can be naturally incorporated into the search process, providing preliminary data for subsequent evaluation and optimization. Combining data-driven learning with symbolic logical reasoning provides a solid and intuitive technical foundation for sequential decision generation in complex industrial scenarios.

[0054] In one embodiment, step 220 may specifically include the following steps: Calculate the cosine similarity of the action sequences contained in each dynamic policy; Based on cosine similarity, a clustering algorithm is used to divide multiple dynamic policies into multiple policy groups.

[0055] Each dynamic policy is mathematically represented for quantitative comparison. To do this, the action sequence contained in each dynamic policy is extracted and transformed into a high-dimensional feature vector. This vector can be constructed based on various features such as the semantic encoding of the action, the type of the operand, and the temporal pattern of the action. Subsequently, the cosine similarity between the feature vectors corresponding to any two dynamic policies is calculated. Cosine similarity is a measure of the difference in direction between two vectors, with a value ranging from -1 to +1. The closer the directions of the action sequences of two policies are in the feature space—that is, the more similar their action types and patterns—the closer the cosine similarity value is to +1, indicating high similarity. Conversely, if the action patterns of two policies differ greatly, their vector directions differ significantly, and the cosine similarity approaches zero or negative values, indicating low similarity. For example, an action sequence primarily focused on finely adjusting valve openings will have a lower cosine similarity in vector representation than another action sequence primarily focused on switching pump operating modes.

[0056] After calculating the cosine similarity between all pairwise dynamic strategies, a matrix describing the similarity relationships within the entire strategy set is obtained. Based on this similarity matrix, a clustering algorithm, such as hierarchical clustering or K-means clustering, is used to automatically divide all dynamic strategies into several clusters, each representing a strategy group. The clustering algorithm automatically groups strategies with high cosine similarity—those whose action trajectories are close to each other in the feature space—into the same group; while strategies with low similarity are grouped into different groups. For example, through clustering, all strategies that primarily control temperature by adjusting the power of heating elements are clustered into one group, while all methods that primarily dissipate heat by adjusting the fluid circulation rate are clustered into another group. This process is entirely data-driven and requires no manual pre-setting of classification rules.

[0057] The grouping method based on cosine similarity calculation and clustering algorithms uses cosine similarity as a metric, effectively capturing the similarity of action sequences in terms of patterns and structure. This makes the grouping results more reflective of the inherent technical differences in strategies. Automated grouping using clustering algorithms avoids exhaustive manual definition and classification of various strategy types in complex industrial scenarios, greatly enhancing the method's versatility and adaptability, enabling it to handle unknown or newly emerging strategy patterns. Through mathematical similarity calculation and clustering, the strategy space is structurally divided, allowing subsequent benefit evaluation and strategy selection within each group to be conducted among more comparable strategies, ensuring fairness in intra-group competition and the effectiveness of the results.

[0058] In one embodiment, step 230 may specifically include the following steps: Discretize the continuous action sequence in the dynamic policy into multiple action blocks and assign dynamic weights to each action block; Calculate the relative benefit of each action block relative to the baseline action block; the relative benefit is determined based on simulated device feedback information, action time, resource consumption, and task completion rate. The payoff value of the dynamic strategy is obtained by weighting the relative payoffs of all action blocks and their corresponding dynamic weights.

[0059] Since a dynamic strategy typically involves a series of consecutive actions, this sequence is discretized into several logically or temporally independent action blocks. Each action block is a sub-segment of the original sequence, containing one or more functionally related consecutive actions. For example, a strategy for controlling chemical production might be discretized into three action blocks: "feeding and preheating stage," "main reaction control stage," and "product output and cooling stage." Each identified action block is assigned a dynamic weight, which can be initially set or subsequently adjusted based on the action block's type, importance in the current state, or historical performance, to characterize the proportion of that action block's contribution to the overall strategy's benefit.

[0060] The independent value of each action block needs to be evaluated. To this end, a baseline action block is defined, which can be a random, average-utility, or predefined reference action sequence unit. For each action block to be evaluated, it is executed individually in the simulation environment, and various data generated during the simulation are collected, collectively referred to as simulation information. This information includes equipment feedback information from the simulation, such as changes in state parameters like temperature and pressure after executing the action block; the execution time of the action block in the simulation; the virtual resources consumed in executing the action block, such as electrical energy and raw materials; and the contribution of the action block to the completion of its assigned subtask. Based on this multi-dimensional simulation information, a predefined evaluation function is used to calculate the relative benefit of the action block relative to the baseline action block. The relative benefit is a scalar value that quantifies the degree of improvement or advantage of the action block in performance compared to the baseline unit; a positive value indicates superiority over the baseline, while a negative value indicates inferiority.

[0061] The overall payoff of the dynamic strategy is calculated by combining the evaluation results of all action blocks. Specifically, the relative payoff of each action block is multiplied by its assigned dynamic weight to obtain its weighted payoff contribution. Then, the weighted payoff contributions of all action blocks are summed, and the total sum is the overall payoff of the dynamic strategy. This process essentially decomposes and reorganizes the long-term payoff of the strategy, obtaining an overall evaluation by assessing and integrating the contributions of each component.

[0062] Discretizing long action sequences into action blocks for independent evaluation effectively reduces the evaluation complexity of long-cycle sequence decisions, enabling detailed analysis of each functional unit in the strategy and helping to identify the strengths and potential bottlenecks. Introducing relative returns to benchmark action blocks as an evaluation unit provides a standardized and comparable value measurement method, allowing action blocks of different functions and types to be measured on a unified scale, enhancing the fairness and consistency of the evaluation. By assigning dynamic weights to each action block and performing weighted summation, the different importance of different action blocks in the overall strategy can be flexibly reflected, making the final calculated strategy return value more closely match the target structure of the actual control task, improving the accuracy of the evaluation results and the targeting of strategy optimization.

[0063] In one embodiment, the following steps may also be included: Obtain the operation logs of electronic devices, and identify and parse high-risk states from the operation logs; Based on the conservative Q-learning algorithm, a safety policy baseline is constructed by adding a conservative regularization term to the loss function of the standard Q-learning algorithm. The conservative regularization term is used to penalize the Q value corresponding to high-risk states and to improve the Q value of predefined safety actions.

[0064] Operation logs accumulated during the long-term operation of electronic devices are retrieved from their control or historical databases. These logs typically record, in chronological order, the control commands received by the device, the environmental and self-state parameters at the time of execution, and the results after execution. Identifying and parsing these logs aims to summarize and define high-risk states. High-risk states refer to device states or state combinations that are strongly correlated with operational failures, severe performance degradation, device alarms, or approaching safety thresholds in historical records. The parsing process can be implemented through statistical analysis, pattern matching, or anomaly detection algorithms. For example, if the state combination of "main motor load rate exceeding 90% for three consecutive minutes" and "coolant inlet temperature exceeding 50 degrees Celsius" significantly increases the probability of subsequent operational failures, this state combination is defined as a high-risk state.

[0065] After identifying high-risk states, this module constructs an intrinsic safety policy baseline based on the conservative Q-learning algorithm. Conservative Q-learning is an improvement on the standard Q-learning algorithm, its core idea being to explicitly avoid high-risk behaviors while optimizing the agent's long-term rewards. Specifically, it adds a conservative regularization term to the loss function of the standard Q-learning algorithm. This conservative regularization term has a dual function. First, it imposes a penalty on the state-action values ​​(Q-values) corresponding to all states identified as high-risk. This means that no matter how high the theoretical reward of an action may be, if it is evaluated under a high-risk state, its final value estimate will be conservatively lowered.

[0066] This regularization term also boosts or biases the Q-values ​​of a set of predefined safety actions. Safety actions are known operations that can guide the device to a more stable or safer state, such as "performing emergency throttling," "switching to standby," or "performing protective shutdown." By incorporating this regularization term into the loss function, the algorithm not only learns how to maximize cumulative rewards during training but is also forced to learn a conservative strategy that naturally tends to avoid high-risk states and prioritizes known safety actions when risks are imminent.

[0067] By automatically learning from historical data and formally defining high-risk states, security knowledge is accumulated in a data-driven manner, eliminating the need for reliance on complex, manually compiled security rule bases by experts. By embedding security constraints as conservative regularization terms into the core optimization objective of the reinforcement learning algorithm, a deep integration of security requirements and task objectives is achieved at the training level. This is more fundamental and effective than simple rule filtering or action masking in the later stages of decision-making, as it fundamentally reshapes the agent's perception of action value, enabling it to inherently possess risk aversion awareness while pursuing performance. The constructed security policy baseline, as an independent value assessment system, can be integrated with the main decision-making process to provide a robust security value reference or constraint for main policy generation. This establishes a reliable security baseline in complex autonomous decision-making, significantly improving the security and reliability of the agent's long-term autonomous operation in unknown or dynamic industrial environments.

[0068] The device control method, apparatus, device, and computer storage medium of this invention acquire device state information of electronic devices collected by multimodal sensors; identify the device state information to obtain multiple atomic states; and achieve a unified and structured representation of the operating conditions of complex devices, transforming the originally heterogeneous and continuous low-level signals into high-level, reasonable symbolic information, laying the foundation for subsequent knowledge-based processing. Multiple state sequences are obtained by querying a pre-constructed state graph based on multiple atomic states. In the state graph, nodes represent atomic states, edges represent the relationships between states, and the weights of the edges represent the historical execution effects of the tasks corresponding to the transition from the predecessor state to the successor state. This is equivalent to encoding historical experience into a queryable knowledge network and performing target reasoning based on the current context. It can proactively plan the evolution path of the device's target state using prior knowledge, realizing a shift from passive response to proactive planning. Multiple state sequences are evaluated using a reinforcement learning strategy to determine a target state sequence. This target state sequence is then parsed into at least one device control task, which instructs the electronic device to perform control actions to achieve the state evolution objective defined by the target state sequence. The abstract target state sequence is then parsed into specific device control tasks and executed, thereby controlling the electronic device according to these tasks. This reduces reliance on external preset instructions or manual programming. By incorporating historical experience into the knowledge base, the autonomous response capability and scientific decision-making of the electronic device in dynamic operating conditions are enhanced, achieving intelligent control of the electronic device.

[0069] The following is combined Figure 3 The following specific examples illustrate this: The system utilizes multimodal sensors to collect equipment status data and automatically generates equipment control tasks, specifically including: 1) Collect equipment status information using sensors such as vision, mechanical, acoustic, and temperature sensors; 2) Perform structured processing on the collected data to generate structured device status information; 3) Automatically identify the structured equipment status information and generate equipment control tasks. These equipment control tasks may include multiple tasks to be executed, each requiring the following steps to be performed separately.

[0070] This includes automatically identifying structured equipment status information and generating equipment control tasks, which may specifically include: First, OCR is used to recognize text, and multiple atomic states are determined based on the recognized text. This realizes the automatic recognition and task decomposition of structured device state information, resulting in multiple atomic states, such as "temperature is 500 degrees".

[0071] Next, a pre-constructed state graph is obtained, and multiple state sequences are dynamically generated based on the identified multiple atomic states. The state graph is used to record the relationships between atomic states and the historical call effects of combined tasks. Nodes in the state graph represent atomic states, edges represent relationships, and the weight of the edges corresponds to the historical call effects of the combined tasks of atomic states.

[0072] Next, a target state sequence is selected from multiple generated state sequences using a reinforcement learning strategy, and a device control task is generated based on the target state sequence. Specifically, PPO / GRPO can be used to select the target state sequence from the multiple generated state sequences. The reward function corresponding to this reinforcement learning strategy includes task completion factor, task time factor, task completion efficiency factor, and resource consumption factor. When the reward value of the target state sequence is higher than a preset value, and the task corresponding to the target state sequence is a new task, a new task discovery is triggered: the target task sequence is abstracted, the abstract information is stored in the task library, and the state graph is updated in reverse.

[0073] High-risk mitigation strategies for generating equipment include: 1) Obtain the device's operation log; 2) Identify and parse the operation logs, and define high-risk states; First, extract the following from the operation log: device ID, operation command, timestamp, running parameters (CPU / memory / temperature / load rate), and operation result (success / failure / alarm).

[0074] High-risk status definition: such as load rate consistently >90%, temperature exceeding safety threshold, illegal command sequence, etc.

[0075] In addition, N consecutive operation records can be used as action behaviors (such as taking the operation sequence within 10 minutes through a sliding window) to characterize the dynamic behavior of the equipment; risk marking can also be performed: embed historical risk frequencies (such as the number of times the equipment has been overloaded in the past hour) into the status.

[0076] Then, a conservative regularization term is added to the standard Q-learning loss to penalize the Q-value of high-risk states: α: Conservative coefficient, controlling the intensity of suppression in high-risk states; a 安全 Safety actions (such as frequency reduction and shutdown protection); S indicates the state, and a indicates the action.

[0077] 3) Generate corresponding strategies: Use operation instructions in high-frequency safe states (such as load <80%) in the statistical log as the initial strategy; directly block instructions in known high-risk states (such as load >95%); dynamically reduce the Q value in potentially risky states (such as load 85%~90%), guide the strategy to select better alternative actions, and guide the device to migrate to a safe state.

[0078] This step utilizes device operation logs to build a security policy baseline through CQL (Conservative Q-Learning) to suppress high-risk actions.

[0079] Multiple dynamic strategies are generated to correspond to the device control tasks, specifically including: Multiple dynamic strategies are grouped, and the benefits of each strategy within a group are calculated. Based on the group benefits, a target dynamic strategy is selected for each group. The device is then controlled based on the target dynamic strategy for each group.

[0080] A dynamic strategy includes multiple consecutive actions. Unlike traditional single-step decision-making, agents can plan long-cycle task chains, which are closer to the continuous operation requirements of industrial scenarios.

[0081] Specifically, multiple dynamic strategies can be generated using the following two methods: Method 1: 1) Construct a causal relationship graph using historical data; where nodes in the causal relationship graph represent device states, actions represent directed edges, and device feedback represents the weights of the edges; Temperature, pressure, etc., are nodes. Action commands (such as "increase power" → "decrease temperature") are edges.

[0082] The causal relationship graph here can be generated using a graph neural network, with training data including state-action pairs and feedback signals from the device.

[0083] 2) Match the device control tasks with nodes in the causal relationship graph, and search the causal relationship graph based on the matched nodes to obtain multiple action paths; each action path includes multiple actions. Each action path corresponds to a dynamic strategy.

[0084] Method 2: Utilizing a trained neural network to determine multiple dynamic strategies. Specifically, the device control task is input into the neural network, which processes the input and outputs multiple dynamic strategies. The neural network is trained based on historical data, learning implicit causal relationships from historical operational data.

[0085] The above-mentioned grouping of multiple dynamic policies can specifically involve grouping action trajectories with similar causal chains into one group, such as obtaining a "high temperature condition" group. Grouping can be based on the cosine similarity of the action sequences, or by clustering the states included in multiple dynamic policies.

[0086] The above-mentioned calculation of returns within an analysis group and selection of a target dynamic strategy can specifically be as follows: 1) Divide the actions in a dynamic policy into blocks to obtain multiple action blocks a. i ; Discretizing continuous actions into action chunks reduces computational dimensionality. Specifically, VQ-VAE can be used to generate discrete action chunks.

[0087] 2) Calculate action block a in this dynamic strategy using the following formula. i Relative returns: Among them, R base R(a) represents the payoff of a random action block. i ) represents action block a i The revenue is determined based on the equipment feedback, action time, resource consumption, and task completion rate simulated by the agent; R group This represents the payoff of an action block in a dynamic strategy.

[0088] The action block a is dynamically determined using the following formula. i Weights: W i+1 =w i +n*RelativeReturn(a i )*Δ wi logπ(a i |s) In the formula, w i Indicates action a i The weight of S represents the current state of the device.

[0089] 3) Calculate the payoff of this dynamic strategy using the following formula: The maximum value of i is the number of action blocks.

[0090] 4) Select the strategy that yields the highest return within the group as the target dynamic strategy for that group.

[0091] Obtain the device feedback corresponding to each target dynamic policy, and optimize the dynamic policy corresponding to the device control task based on the device feedback. Specifically, the neural network that generates the dynamic policy can be optimized.

[0092] Therefore, embodiments of the present invention, based on the task autonomous generation and knowledge evolution mechanism of reinforcement learning, utilize reinforcement learning strategies to evaluate and select the optimal target state sequence from multiple candidate state sequences automatically generated by the system, and then parse it into a specific device control task. Crucially, when the device control task is identified as a new, unrecorded task, the system can automatically abstract the task and update its key information in a pre-constructed state graph. This process enables the agent to autonomously perceive new tasks and dynamically expand its knowledge base, laying an adaptive foundation for device control in "wilderness" environments lacking complete prior knowledge.

[0093] An intrinsic safety mechanism based on conservative reinforcement learning identifies and defines clear high-risk states by analyzing historical operation logs of the equipment. A conservative Q-learning algorithm is then used to construct a safety policy baseline. This algorithm systematically penalizes the value estimation of actions associated with high-risk states by introducing a conservative regularization term into the standard training objective, while simultaneously increasing the value of known safe actions. This enables the agent to intrinsically avoid known high-risk operations at the policy level and suppress the value of suboptimal actions in potentially risky states, thereby guiding the equipment to migrate to a safer state and ensuring the safety of the control process from the source of decision-making.

[0094] To meet the practical needs of continuous operation in industrial scenarios, multiple long-cycle dynamic strategies consisting of several consecutive actions are dynamically generated for a given equipment control task by using pre-built causal relationship graphs or trained neural network models. This changes the limitations of traditional single-step decision-making, enabling intelligent agents to perform multi-step sequential planning, thus better aligning with complex industrial process control.

[0095] To address the computational challenges posed by the vast policy search space, multiple generated dynamic policies are grouped based on the similarity of their action trajectories. Then, during intra-group evaluation, the continuous action sequence of each dynamic policy is discretized into multiple action blocks for computation. By calculating the relative payoff of each action block and performing a weighted ensemble, the target dynamic policy within each group is efficiently determined. This grouping and block-based approach significantly reduces the computational complexity of evaluating long-sequence policies.

[0096] By executing a target dynamic policy and acquiring real feedback from industrial equipment, the model generating the dynamic policy is continuously optimized using this feedback data within a reinforcement learning framework. This closed-loop learning process enables autonomous iteration and performance improvement of the control policy, allowing the agent to continuously adapt to changes in complex environments and effectively handle emerging tasks, ultimately achieving the core goal of continuous autonomous evolution in dynamic and uncertain environments.

[0097] Figure 4This is a schematic diagram of a device structure provided in an embodiment of the present invention. Figure 4 As shown, the device may include an acquisition module 410, a generation module 420, an identification module 430, a matching module 440, and a determination module 450.

[0098] The acquisition module 410 is used to acquire device status information of electronic devices collected by multimodal sensors; The identification module 420 is used to identify the device status information and obtain multiple atomic states; The query module 430 is used to query a pre-built state graph based on multiple atomic states to obtain multiple state sequences. The nodes of the state graph represent atomic states, the edges of the state graph represent the relationships between states, and the weight of the edge represents the historical execution effect of the task corresponding to the transition from the predecessor state to the successor state. Evaluation module 440 is used to evaluate multiple state sequences through a reinforcement learning strategy and determine the target state sequence from the multiple state sequences; The parsing module 450 is used to parse the target state sequence into at least one device control task, wherein the device control task is used to instruct the electronic device to perform control actions to achieve the state evolution target defined by the target state sequence. The control module 460 is used to control the electronic equipment according to the equipment control task.

[0099] In one embodiment, the evaluation module 440 is specifically used for: The reward value for each sequence of states is obtained by evaluating multiple state sequences using a reinforcement learning strategy. The state sequence with the highest reward value is determined as the target state sequence.

[0100] In one embodiment, the device may further include: The processing module is used to abstract the target state sequence and generate abstract task information when the reward value of the target state sequence is higher than a preset threshold and the abstract task information corresponding to the target state sequence is not stored in the task library. The storage module is used to store abstract task information into the task library; The update module is used to update the weights of the corresponding edges in the state graph based on the device feedback information, which is generated by the electronic device after it performs a device control task.

[0101] In one embodiment, the control module 460 is specifically used for: Multiple dynamic strategies are generated based on the equipment control task. Each dynamic strategy contains a sequence of actions consisting of multiple consecutive actions. Multiple dynamic strategies are grouped according to the similarity of their action trajectories to obtain multiple strategy groups; Within each strategy group, the benefit value of the dynamic strategy is determined based on the simulation information of the dynamic strategy; the simulation information includes: simulated device feedback information, action time, resource consumption, and task completion rate; Within each strategy group, the dynamic strategy with the highest return value is identified as the target dynamic strategy; Controlling electronic devices based on target dynamic strategies.

[0102] In one embodiment, the control module 460 is specifically used for: The device control task is matched with nodes in a pre-built causal relationship graph, where nodes represent device states, edges represent actions, and edge weights represent device feedback. The pre-built causal relationship graph is obtained by training a graph neural network on historical state information, historical action sequences, and device feedback signals. Path search is performed in the causal relationship graph to obtain multiple action paths; Each action path is treated as a dynamic policy to obtain multiple dynamic policies.

[0103] In one embodiment, the control module 460 is specifically used for: Calculate the cosine similarity of the action sequences contained in each dynamic policy; Based on cosine similarity, a clustering algorithm is used to divide multiple dynamic policies into multiple policy groups.

[0104] In one embodiment, the control module 460 is specifically used for: Discretize the continuous action sequence in the dynamic policy into multiple action blocks and assign dynamic weights to each action block; Calculate the relative benefit of each action block relative to the baseline action block; the relative benefit is determined based on simulated device feedback information, action time, resource consumption, and task completion rate. The payoff value of the dynamic strategy is obtained by weighting the relative payoffs of all action blocks and their corresponding dynamic weights.

[0105] In one embodiment, the identification module 420 is specifically used for: The equipment status information is processed in a structured manner to generate structured equipment status information; The state information of structured equipment is identified and decomposed into multiple atomic states.

[0106] In one embodiment, the acquisition module 410 is further configured to acquire the operation log of the electronic device and identify and parse the high-risk state from the operation log; The device may also include: The module is used to build a safety policy baseline based on the conservative Q-learning algorithm by adding a conservative regularization term to the loss function of the standard Q-learning algorithm. The conservative regularization term is used to penalize the Q value corresponding to high-risk states and improve the Q value of predefined safety actions.

[0107] The device control method, apparatus, device, and computer storage medium of this invention acquire device state information of electronic devices collected by multimodal sensors; identify the device state information to obtain multiple atomic states; and achieve a unified and structured representation of the operating conditions of complex devices, transforming the originally heterogeneous and continuous low-level signals into high-level, reasonable symbolic information, laying the foundation for subsequent knowledge-based processing. Multiple state sequences are obtained by querying a pre-constructed state graph based on multiple atomic states. In the state graph, nodes represent atomic states, edges represent the relationships between states, and the weights of the edges represent the historical execution effects of the tasks corresponding to the transition from the predecessor state to the successor state. This is equivalent to encoding historical experience into a queryable knowledge network and performing target reasoning based on the current context. It can proactively plan the evolution path of the device's target state using prior knowledge, realizing a shift from passive response to proactive planning. Multiple state sequences are evaluated using a reinforcement learning strategy to determine a target state sequence. This target state sequence is then parsed into at least one device control task, which instructs the electronic device to perform control actions to achieve the state evolution objective defined by the target state sequence. The abstract target state sequence is then parsed into specific device control tasks and executed, thereby controlling the electronic device according to these tasks. This reduces reliance on external preset instructions or manual programming. By incorporating historical experience into the knowledge base, the autonomous response capability and scientific decision-making of the electronic device in dynamic operating conditions are enhanced, achieving intelligent control of the electronic device.

[0108] Figure 5 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention is shown.

[0109] An electronic device may include a processor 301 and a memory 302 storing computer program instructions.

[0110] Specifically, the processor 301 may include a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of the present invention.

[0111] Memory 302 may include mass storage for data or instructions. For example, and not limitingly, memory 302 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. In one instance, memory 302 may include removable or non-removable (or fixed) media, or memory 302 may be non-volatile solid-state memory. Memory 302 may be internal or external to the integrated gateway disaster recovery device.

[0112] In one instance, memory 302 may be read-only memory (ROM). In one instance, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or flash memory, or a combination of two or more of these.

[0113] Memory 302 may include read-only memory (ROM), random access memory (RAM), disk storage media device, optical storage media device, flash memory device, electrical, optical, or other physical / tangible memory storage device. Therefore, generally, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to one aspect of this disclosure.

[0114] The processor 301 reads and executes computer program instructions stored in the memory 302 to achieve... Figure 5 The device control method in the illustrated embodiment.

[0115] Furthermore, in conjunction with the device control methods described in the above embodiments, this invention can be implemented using a computer storage medium. This computer storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the device control methods described in the above embodiments.

[0116] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements any of the device control methods described in the above embodiments.

[0117] It should be clarified that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of the present invention.

[0118] The functional blocks shown in the above-described structural diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this invention are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, read-only memory (ROM), flash memory, erasable read-only memory (EROM), floppy disks, compact disc read-only memory (CD-ROM), optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0119] It should also be noted that the exemplary embodiments mentioned in this invention describe methods or systems based on a series of steps or apparatus. However, this invention is not limited to the order of the steps described above; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0120] The aspects of this disclosure have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by special-purpose hardware performing the specified functions or actions, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0121] The above description is merely a specific embodiment of the present invention. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the protection scope of the present invention.

Claims

1. A device control method, characterized in that, The method includes: Acquire device status information of electronic devices collected by multimodal sensors; The device status information is identified to obtain multiple atomic states; Based on the multiple atomic states, a pre-constructed state graph is queried to obtain multiple state sequences. The nodes of the state graph represent the atomic states, the edges of the state graph represent the relationships between states, and the weights of the edges represent the historical execution effects of the tasks corresponding to the transition from the predecessor state to the successor state. The multiple state sequences are evaluated using a reinforcement learning strategy to determine the target state sequence from the multiple state sequences; The target state sequence is parsed into at least one device control task, wherein the device control task is used to instruct the electronic device to perform control actions to achieve the state evolution target defined by the target state sequence; The electronic device is controlled according to the device control task.

2. The method according to claim 1, characterized in that, The step of evaluating the plurality of state sequences using a reinforcement learning strategy and determining the target state sequence from the plurality of state sequences includes: The multiple state sequences are evaluated using a reinforcement learning strategy to obtain a reward value for each of the multiple state sequences; The state sequence with the highest reward value is determined as the target state sequence.

3. The method according to claim 1, characterized in that, After evaluating the plurality of state sequences using a reinforcement learning strategy and determining the target state sequence from the plurality of state sequences, the method further includes: If the reward value of the target state sequence is higher than a preset threshold and the abstract task information corresponding to the target state sequence is not stored in the task library, the target state sequence is abstracted to generate abstract task information. The abstract task information is stored in the task library; Based on the device feedback information, the weights of the corresponding edges in the state graph are updated. The device feedback information is generated by the electronic device after executing the device control task.

4. The method according to claim 1, characterized in that, The step of controlling the electronic device according to the device control task includes: Multiple dynamic strategies are generated based on the device control task, wherein each dynamic strategy contains an action sequence consisting of multiple consecutive actions; The multiple dynamic strategies are grouped according to the similarity of their action trajectories to obtain multiple strategy groups; Within each strategy group, the benefit value of the dynamic strategy is determined based on the simulation information of the dynamic strategy; the simulation information includes: simulated device feedback information, action time, resource consumption, and task completion rate; Within each strategy group, the dynamic strategy with the highest return value is identified as the target dynamic strategy; The electronic device is controlled based on the target dynamic strategy.

5. The method according to claim 4, characterized in that, The generation of multiple dynamic strategies based on the device control task includes: The device control task is matched with nodes in a pre-constructed causal relationship graph, where nodes in the causal relationship graph represent device states, edges in the causal relationship graph represent actions, and the weights of the edges represent device feedback; the pre-constructed causal relationship graph is obtained by training a graph neural network on historical state information, historical action sequences, and device feedback signals. Path search is performed in the causal relationship graph to obtain multiple action paths; Each of the aforementioned action paths is treated as a dynamic policy to obtain multiple dynamic policies.

6. The method according to claim 4, characterized in that, The process of grouping the multiple dynamic strategies according to the similarity of their action trajectories results in multiple strategy groups, including: Calculate the cosine similarity of the action sequences contained in each dynamic policy; Based on the cosine similarity, a clustering algorithm is used to divide the multiple dynamic strategies into multiple strategy groups.

7. The method according to claim 4, characterized in that, Determining the payoff value of each dynamic strategy within each strategy group includes: The continuous action sequence in the dynamic strategy is discretized into multiple action blocks, and a dynamic weight is assigned to each action block. Calculate the relative benefit of each action block relative to the baseline action block; the relative benefit is determined based on simulated device feedback information, action time, resource consumption, and task completion rate. The payoff value of the dynamic strategy is obtained by performing a weighted calculation based on the relative payoff of all action blocks and their corresponding dynamic weights.

8. The method according to claim 1, characterized in that, The process of identifying the device state information yields multiple atomic states, including: The device status information is processed into a structured form to generate structured device status information. The state information of the structured device is identified and decomposed into multiple atomic states.

9. The method according to claim 1, characterized in that, The method further includes: Obtain the operation log of the electronic device, and identify and parse high-risk states from the operation log; Based on the conservative Q-learning algorithm, a safety policy baseline is constructed by adding a conservative regularization term to the loss function of the standard Q-learning algorithm. The conservative regularization term is used to penalize the Q value corresponding to the high-risk state and to improve the Q value of the predefined safety action.

10. A device control apparatus, characterized in that, The device includes: The acquisition module is used to acquire device status information of electronic devices collected by multimodal sensors; The identification module is used to identify the device status information and obtain multiple atomic states; The query module is used to query a pre-constructed state graph based on the multiple atomic states to obtain multiple state sequences. The nodes of the state graph represent the atomic states, the edges of the state graph represent the relationships between states, and the weights of the edges represent the historical execution effects of the tasks corresponding to the transition from the predecessor state to the successor state. An evaluation module is used to evaluate the plurality of state sequences using a reinforcement learning strategy and determine a target state sequence from the plurality of state sequences; A parsing module is used to parse the target state sequence into at least one device control task, wherein the device control task is used to instruct the electronic device to perform control actions to achieve the state evolution target defined by the target state sequence; The control module is used to control the electronic device according to the device control task.

11. An electronic device, characterized in that, The device includes: a processor and a memory storing computer program instructions; the processor reads and executes the computer program instructions to implement the device control method as described in any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, The computer storage medium stores computer program instructions, which, when executed by a processor, implement the device control method as described in any one of claims 1-9.

13. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the device control method as described in any one of claims 1-9.

Images