Equipment control method and system, equipment and storage medium

Abstract

The invention provides a device control method and system, a device and a storage medium, and the method comprises the steps: an edge side device receives multi-modal input data sent by each end side device; the edge side device performs reasoning based on the multi-modal input data sent by each end side device, generates future action prediction information of each end side device, and sends the future action prediction information of each end side device to each end side device; and each end-side device updates an annular action cache queue in the end-side device based on the future action prediction information, determines a current control instruction from the annular action cache queue according to the current moment, and executes the current control instruction. According to the method, a calculation-intensive reasoning process can be separated from an execution process with an extremely high real-time requirement, so that continuity and smoothness of action output are realized, and motion discontinuity is avoided fundamentally. And meanwhile, the robustness and the stability of the control process can be remarkably improved.

Description

Technical Field

[0001] This application relates to the field of equipment control technology, and more specifically, to an equipment control method, system, device, and storage medium. Background Technology

[0002] With the deep integration of artificial intelligence and robotics, embodied AI, as a key pathway to achieving autonomous perception, decision-making, and action in robots, is receiving widespread attention. Embodied AI unifies vision, language, and action (VLA) within a single model framework. It perceives the environmental state in real time through multimodal sensors (such as cameras, microphones, torque / force control sensors, inertial measurement units, and tactile sensors) and generates continuous action sequences using pre-trained large-scale models or policy networks to drive robots to complete complex tasks.

[0003] In existing technologies, the inference deployment of embodied intelligent models can adopt a local area network edge inference architecture, where edge servers are deployed on the same site as inference service nodes, and the end side communicates with them in the form of a lightweight client to obtain inference results.

[0004] However, while this edge inference method reduces network latency, it still uses a serial mechanism of "request-execute" at the action delivery level. It does not solve the last-mile problem of "multi-step action overlap processing" and "ultra-high frequency stable scheduling". That is, there is still a lack of time consistency calibration between action sequences returned by different inference cycles. When switching, speed or acceleration jumps are easy to occur, resulting in trajectory "stuttering". In addition, the queue buffer design is crude and lacks fine-grained time synchronization and prediction interpolation mechanisms. When faced with packet loss or inference delay, it is easy to produce window periods or repeated execution, affecting control continuity and motion smoothness. Summary of the Invention

[0005] The purpose of this application is to address the shortcomings of the prior art by providing a device control method, system, device, and storage medium to solve the problems in the prior art where there is a lack of timing consistency calibration between action sequences returned by different inference cycles, and speed or acceleration jumps are prone to occur during switching, resulting in trajectory "stuttering" and poor control continuity and motion smoothness.

[0006] To achieve the above objectives, the technical solutions adopted in the embodiments of this application are as follows: In a first aspect, one embodiment of this application provides a device control method applied to a device inference system, the device inference system comprising: an edge-side device disposed in a local area network and multiple end-side devices, the method comprising: The edge device receives multimodal input data sent by each of the end devices. The multimodal input data includes multimodal sensor data and the end timestamp corresponding to the multimodal sensor data. The edge device performs inference based on the multimodal input data sent by each of the end devices, generates future action prediction information for each of the end devices, and sends the future action prediction information of each of the end devices to each of the end devices. The future action prediction information includes: action information of the end device in multiple future time steps and timestamps corresponding to each action information. Each of the aforementioned end-side devices updates the circular action buffer queue in the end-side device based on the future action prediction information, and determines the current control command from the circular action buffer queue according to the current time, and executes the current control command.

[0007] In one possible implementation, the step of inferring based on the multimodal input data sent by each of the end-side devices to generate future action prediction information for each of the end-side devices includes: The global timestamp of the multimodal sensor data is determined based on the end-side timestamp corresponding to the multimodal sensor data. Based on the global timestamps of the multimodal sensor data, determine the multimodal state vectors at multiple time points; Based on the multimodal state vectors at each time point and the pre-trained inference model, the action information of the edge device is generated for multiple time steps in the future. Add corresponding timestamps to the action information of the terminal device at multiple future time steps to obtain the future action prediction information.

[0008] In one possible implementation, updating the circular action cache queue in the endpoint device based on the future action prediction information includes: Based on the timestamp corresponding to the action information, determine whether there is historical action information corresponding to the time point in the circular action cache queue in the terminal device; If it exists, the circular action cache queue in the terminal device is updated according to the action information and the historical action information; If it does not exist, the circular action cache queue in the terminal device is updated according to the action information.

[0009] In one possible implementation, updating the circular action buffer queue in the endpoint device based on the action information and the historical action information includes: The action information and the historical action information are weighted and fused to obtain the current action information, and the current action information is filled into the corresponding position of the circular action cache queue in the terminal device.

[0010] In one possible implementation, the weighted fusion of the action information and the historical action information to obtain the current action information includes: Determine the current scheduling time and the relative time freshness of the action information at the current scheduling time; The uncertainty of the action information is determined; Based on the relative time freshness and the uncertainty, the action information and the historical action information are weighted and fused to obtain the current action information.

[0011] In one possible implementation, the step of weightedly fusing the action information and the historical action information based on the relative time freshness and the uncertainty to obtain the current action information includes: Based on the relative time freshness and the uncertainty, the first weight and the second weight are calculated; Calculate the product of the first weight and the action information to obtain the first product result; Calculate the product of the second weight and the historical action information to obtain the second product result; The sum of the first product result and the second product result is calculated and used as the current action information.

[0012] In one possible implementation, determining the current control instruction from the circular action buffer queue based on the current moment includes: Based on the current time, determine whether there is action information corresponding to the current time in the circular action buffer queue; If so, then generate the current control command based on the action information corresponding to the current moment; If not, the current control command is generated by interpolation based on the action information corresponding to the previous moment and / or the next moment.

[0013] Secondly, another embodiment of this application provides a device inference system, the device inference system comprising: an edge-side device disposed in a local area network and a plurality of end-side devices, the device inference system being used to execute the device control method described in the first aspect.

[0014] Thirdly, another embodiment of this application provides an electronic device, including: a processor, a storage medium, and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of any of the methods described in the first aspect above.

[0015] Fourthly, another embodiment of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of any of the methods described in the first aspect above.

[0016] The beneficial effects of this application are as follows: By receiving multimodal input data sent by each end-side device through edge-side devices, and performing inference based on the multimodal input data sent by each end-side device to generate future action prediction information for each end-side device, this future action prediction information is sent to each end-side device. Each end-side device then updates its circular action buffer queue based on the future action prediction information, and determines and executes the current control command from the circular action buffer queue according to the current time. This separates the computationally intensive inference process from the execution process, which has extremely high real-time requirements, thereby achieving continuity and smoothness in action output and fundamentally avoiding motion discontinuity. Simultaneously, it significantly improves the robustness and stability of the control process. Furthermore, it has the advantages of low latency, easy scalability, and support for stable ultra-high frequency scheduling. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A schematic diagram of the structure of a device control system provided in an embodiment of this application; Figure 2 A schematic flowchart of a device control method provided in an embodiment of this application; Figure 3 This is a schematic flowchart illustrating the process of generating future action prediction information for each end-side device in the device control method provided in the embodiments of this application. Figure 4 This is a schematic flowchart illustrating the process of updating the circular action buffer queue in the end-side device in the device control method provided in the embodiments of this application. Figure 5A schematic flowchart illustrating the process of obtaining current action information in the device control method provided in this application embodiment; Figure 6 Another flowchart illustrating the process of obtaining current action information in the device control method provided in this application embodiment; Figure 7 A flowchart illustrating the process of determining the current control command in the device control method provided in this application embodiment; Figure 8 This is a schematic diagram of the electronic device structure provided in an embodiment of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0020] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0021] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0022] In existing technologies, the inference deployment of embodied intelligent models can adopt a local area network edge inference architecture, where edge servers are deployed on the same site as inference service nodes, and the end side communicates with them in the form of a lightweight client to obtain inference results.

[0023] However, while this edge inference method reduces network latency, it still uses a serial mechanism of "request-execute" at the action delivery level. It does not solve the last-mile problem of "multi-step action overlap processing" and "ultra-high frequency stable scheduling". That is, there is still a lack of time consistency calibration between action sequences returned by different inference cycles. When switching, speed or acceleration jumps are easy to occur, resulting in trajectory "stuttering". In addition, the queue buffer design is crude and lacks fine-grained time synchronization and prediction interpolation mechanisms. When faced with packet loss or inference delay, it is easy to produce window periods or repeated execution, affecting control continuity and motion smoothness.

[0024] Based on the aforementioned problems, this application proposes a device control method. The method involves an edge device receiving multimodal input data from each end-side device, performing inference based on this data to generate future action prediction information for each end-side device, and sending this prediction information to each device. Each end-side device then updates its circular action buffer queue based on this prediction information. Finally, it determines and executes the current control command from the circular action buffer queue, enabling continuous and smooth action output from the end-side devices, avoiding action jumps, and improving control robustness. This method offers advantages such as low latency, high robustness, and support for stable ultra-high frequency scheduling.

[0025] First, the equipment control system involved in the equipment control method provided in the embodiments of this application will be described in detail.

[0026] Figure 1 This is a schematic diagram of a device control system provided in an embodiment of this application, with reference to... Figure 1 As shown, the equipment control system includes: edge-side devices and multiple end-side devices located in the local area network.

[0027] Among them, the end-side device can be an onboard controller deployed on the robot body, such as an embedded industrial computer or a small computing unit (e.g., NVIDIA Jetson Xavier / Orin).

[0028] Among them, the edge-side device can be a high-performance server node located in the same local area network as each end-side device, such as an edge computing host deployed in the same local area network as each end-side device.

[0029] For example, an inference client can be deployed on the edge device, which is used for multimodal input data acquisition and lightweight preprocessing, timestamp alignment, state estimation, action reconstruction and caching, millisecond-level delivery, and safe amplitude limiting / anti-jitter control. An inference server can be deployed on the edge device, which is used to load the trained VLA model, receive the packaged multimodal input data from the edge device, perform one inference to return dozens of future actions (points / trajectory segments / command sequences), and provide batch / streaming interfaces.

[0030] Optionally, a synchronization clock can also be deployed in the edge devices and multiple end devices in the device inference system. The synchronization clock establishes a unified and stable global time coordinate system through the PTP protocol stack that supports hardware timestamps, ensuring that the time tags of all subsequent actions can be aligned.

[0031] For example, the synchronization clock is a high-precision, system-wide unified time coordinate system established through the PTP (Precision Time Protocol, IEEE 1588) protocol. The synchronization clock includes a master clock and several slave clocks, with the edge server acting as the PTP master clock and each end device acting as a slave clock.

[0032] Specifically, edge devices and multiple end devices periodically exchange time synchronization messages, and within the interval between two synchronizations, a micro-synchronization strategy is adopted to maintain sub-millisecond time consistency, ensuring the accuracy of the time anchor point for subsequent action scheduling. At the same time, a local drift estimator (LDE) is constructed to model frequency offset based on historical deviations.

[0033] The device control method provided in this application will be described in detail below with reference to several embodiments.

[0034] Figure 2 This is a schematic flowchart of a device control method provided in an embodiment of this application, referring to... Figure 2 As shown, the execution entity of this method can be the aforementioned device inference system, and the method includes: S201, The edge-side device receives multimodal input data sent by each end-side device.

[0035] It is understandable that each edge device can collect raw data from the robot's camera, microphone, torque sensor, inertial measurement unit (IMU) in real time, and after preprocessing such as image encoding, speech vectorization and feature extraction, package it into message frames as multimodal sensor data, attach an edge timestamp generated by a synchronization clock, and send the multimodal sensor data and the corresponding edge timestamp to the edge device.

[0036] For example, each end-side device can send multimodal sensor data and the corresponding end-side timestamp to the edge-side device via the local area network UDP / TCP protocol.

[0037] Optionally, edge devices can receive multimodal input data sent by end devices via LAN UDP / TCP protocol.

[0038] The multimodal input data includes: multimodal sensor data and the corresponding end-side timestamps of the multimodal sensor data.

[0039] For example, multimodal sensor data consists of raw or preprocessed information reflecting the state of the environment or the sensor's own state, collected by various types of sensors deployed on an edge device. Multimodal sensor data includes data from multiple different modalities, providing complementary and more comprehensive state awareness.

[0040] For example, multimodal sensor data may include: images or video frames captured by a camera, sound signals captured by a microphone, contact force information fed back by a joint torque sensor, contact force information fed back by a tactile sensor, acceleration provided by an IMU, angular velocity provided by an IMU, point clouds from a lidar, distance sensors (e.g., ultrasonic, infrared), temperature, humidity, etc.

[0041] For example, the edge timestamp corresponding to multimodal sensor data refers to the time stamp added by the local clock system of the edge device at the moment when each multimodal sensor data is collected or packaged.

[0042] S202, The edge device performs inference based on the multimodal input data sent by each end device, generates future action prediction information for each end device, and sends the future action prediction information of each end device to each end device.

[0043] Optionally, after receiving multimodal input data from each end-side device, the edge-side device can invoke the loaded inference model and perform batch inference based on the multimodal input data sent by each end-side device, and output the action information of each end-side device at multiple future time steps. Simultaneously, a corresponding timestamp is added to the action information of each end-side device at multiple future time steps.

[0044] For example, taking the case where an edge device receives multimodal input data sent by an end device, the edge device receives the multimodal input data sent by the end device and performs forward inference to generate action information of the end device in multiple future time steps. At the same time, based on a synchronization clock, a unique time index is assigned to the action information of the end device in multiple future time steps.

[0045] For example, when an edge device receives multimodal input data from multiple end devices, the edge device can perform parallel batch inference using multiple threads to output the action information of each end device at multiple future time steps. Simultaneously, a corresponding timestamp is added to the action information of each end device at these future time steps.

[0046] Optionally, the edge device sends the future action prediction information of each end device to each end device.

[0047] The future action prediction information includes: action information of the terminal device at multiple time steps in the future, and the timestamp corresponding to each action information.

[0048] Specifically, the action information of the edge device in the future multiple time steps refers to a series of action instructions that the edge device predicts should be executed by the edge device in the future.

[0049] For example, depending on the task, the motion information of the end-side device in the future multiple time steps can be represented as: robot joint angle sequence, end effector trajectory point sequence, force control command sequence, and discrete behavior command stream sequence.

[0050] Specifically, the timestamp corresponding to each action information refers to the target execution time of each action information on the global timeline, used to indicate at what moment each action information should be executed.

[0051] S203. Each end-side device updates the circular action buffer queue in the end-side device based on the future action prediction information, and determines the current control command from the circular action buffer queue according to the current time, and executes the current control command.

[0052] Optionally, each end-side device may pre-maintain a circular motion buffer queue, which is updated when future motion prediction information is received, and the current control command is determined from the circular motion buffer queue according to the current time and executed.

[0053] The circular motion buffer queue is a fixed-length circular queue that organizes motion information for a future period of time in chronological order. This motion information can include, for example, position, acceleration, and velocity.

[0054] Here, the current time refers to the target time for the next action to be performed by the end-side device. Specifically, the current time refers to the instantaneous reading of the slave synchronization clock running on the end-side device after PTP synchronization and local drift compensation by the local drift estimator.

[0055] For example, the local clock of the edge device is determined by a global synchronization protocol (such as PTP), resulting in a unique and precise global time point that is fully aligned with the edge device and other edge devices. By determining the current control command from the circular action buffer queue at the current moment, it is possible to ensure the stable issuance of the current control command at the millisecond level and the smooth transition of actions.

[0056] Specifically, each element in the circular action buffer queue corresponds to a timestamp, thus forming a time-aligned action trajectory segment. The length of the circular action buffer queue covers an action window for a future period of time.

[0057] For example, when each end-side device sends multimodal sensor data and the corresponding end-side timestamp to the edge-side device, it can open a listening port to receive future action prediction information from one or more end-side devices, and after parsing it, update the circular action cache queue in the end-side device.

[0058] For example, the data organization of the circular action cache queue can be a one-dimensional wake-up buffer with time index. The circular action cache queue is arranged in chronological order. Specifically, the circular action cache queue can be a one-dimensional array, and each element is an action state structure containing multiple fields.

[0059] For example, each end device can run a high-frequency scheduling thread. The high-frequency scheduling thread is driven by the local synchronous clock. At the beginning of each scheduling cycle, it determines the current time, determines the current control instruction from the circular action buffer queue based on the current time, and executes the current control instruction.

[0060] In this embodiment, the edge device receives multimodal input data sent by each end-side device, and performs inference based on this data to generate future action prediction information for each end-side device. This prediction information is then sent to each end-side device, which updates its circular action buffer queue based on this information. The current control command is then determined from the buffer queue and executed. This separation of the computationally intensive inference process from the real-time-critical execution process ensures continuous and smooth action output, fundamentally preventing discontinuous motion. Furthermore, it significantly improves the robustness and stability of the control process. Additionally, it offers advantages such as low latency, easy scalability, and support for stable ultra-high frequency scheduling.

[0061] In one possible implementation, Figure 3 This is a schematic flowchart illustrating the process of generating future action prediction information for each end-side device in the device control method provided in the embodiments of this application, with reference to... Figure 3 As shown, in step S202 above, inference is performed based on the multimodal input data sent by each end-side device to generate future action prediction information for each end-side device, including: S301. Determine the global timestamp of the multimodal sensor data based on the end-side timestamp corresponding to the multimodal sensor data.

[0062] Optionally, after obtaining the multimodal input data, the global timestamp of the multimodal sensor data can be determined based on the end-side timestamp corresponding to the multimodal sensor data in the multimodal input data and the pre-established unified global time reference.

[0063] The pre-established unified global time base is the aforementioned synchronization clock. The global timestamp refers to the timestamp under the PTP master clock.

[0064] For example, each end-side device periodically performs PTP synchronization with the edge-side device to obtain its own clock offset and frequency drift. When the end-side device sends multimodal sensor data, it carries the end-side timestamp corresponding to the multimodal sensor data. After receiving the multimodal sensor data, the edge-side device converts it into a global timestamp of the multimodal sensor data according to the pre-stored clock offset and frequency drift of the end-side device.

[0065] S302. Based on the global timestamps of the multimodal sensor data, determine the multimodal state vectors at multiple time points.

[0066] Optionally, after obtaining the global timestamp of the multimodal sensor data, the multimodal sensor data can be sorted according to the global timestamp and cached within a preset sliding time window to obtain a time-aligned data pool.

[0067] Optionally, multiple timestamps can be determined based on global timestamps as the basis for modeling. These multiple timestamps are multiple consecutive global time points.

[0068] In one example, multiple moments can be sampled at equal intervals according to a preset time interval.

[0069] In another example, the sampling interval can be dynamically adjusted based on key events of the task, allowing sampling to be performed at multiple points in time.

[0070] In another example, multiple moments can be obtained by performing fixed sampling based on a preset time window of fixed length.

[0071] Optionally, after obtaining multiple time points, for each time point, the modal data can be retrieved from the time-aligned data pool, and the multimodal state vector at each time point can be constructed.

[0072] For example, taking a specific moment as an example, we can find the modal data near that moment from the time-aligned data pool through a time proximity matching strategy, encode and extract features from each modal data to obtain the feature vectors corresponding to each modal data, and then concatenate or fuse the feature vectors corresponding to each modal data into a multimodal state vector at that moment through splicing or fusion.

[0073] Among them, the multimodal state vectors at multiple time points are a set of ordered multimodal state vectors, corresponding to multiple historical time steps.

[0074] S303. Based on the multimodal state vectors at each time point and the pre-trained inference model, generate action information of the edge device at multiple time steps in the future.

[0075] Optionally, the multimodal state vectors at each time step can be encoded into an input sequence, and the input sequence can be input into a pre-trained inference model. The inference model can then make predictions to obtain the action information of the edge device at multiple time steps in the future.

[0076] In this context, the motion information of the end-device at multiple future time steps can be represented as a sequence of future actions. The motion information at each time step can include: joint angle vectors, end-effector pose, force control commands, and discrete behavior commands. The motion information at each time step can be used as the original motion value or as the motion increment.

[0077] Optionally, the inference model can also simultaneously predict uncertainty scores for the actions of the edge device at multiple future time steps. The uncertainty score is used to indicate the reliability of the action information.

[0078] The inference model can be, for example, a Transformer-based VLA model or an LSTM / GRU sequence model.

[0079] S304. Add corresponding timestamps to the action information of the end-side device at multiple future time steps to obtain future action prediction information.

[0080] Optionally, after obtaining action information for multiple future time steps, a corresponding target execution time can be assigned to each action according to a preset action scheduling frequency, which serves as the corresponding timestamp, to obtain future action prediction information.

[0081] By using the edge-side timestamps corresponding to the multimodal sensor data, the global timestamp of the multimodal sensor data is determined. Then, using the global timestamps of the multimodal sensor data, the multimodal state vectors at multiple time points are determined. Based on the multimodal state vectors at each time point and the pre-trained inference model, the action information of the edge device at multiple time steps in the future is generated. Corresponding timestamps are added to the action information of the edge device at multiple time steps in the future to obtain future action prediction information. This realizes the leap from "fragmented perception-isolated decision-making" to a closed loop of "time-consistent perception-prediction-execution", thereby ensuring the continuity, smoothness, accuracy and robustness of robot actions.

[0082] In one possible implementation, Figure 4 A schematic flowchart illustrating the updating of the circular action buffer queue in the end-side device in the device control method provided in this application embodiment is shown below. Figure 4 As shown, in S203 above, updating the circular action buffer queue in the terminal device based on future action prediction information includes: S401. Based on the timestamp corresponding to the action information, determine whether there is historical action information corresponding to the time point in the circular action buffer queue in the terminal device.

[0083] Optionally, the system can search the circular action cache queue in the terminal device for historical action information corresponding to the timestamp of the action information.

[0084] Historical action information refers to action information that has been stored in the circular action cache queue.

[0085] S402. If it exists, update the circular action cache queue in the terminal device according to the action information and historical action information.

[0086] Optionally, if the circular action cache queue contains historical action information at a corresponding time point, the circular action cache queue in the terminal device can be updated based on the action information and the historical action information.

[0087] In one example, historical motion information can be retrieved from the circular motion cache queue, and the motion information can be compared with the historical motion information. The motion information with smaller motion amplitude compared with the historical motion information can be updated to the circular motion cache queue, which can achieve a smoother trajectory transition, improve motion smoothness, and avoid abrupt motion changes.

[0088] In another example, historical action information can be retrieved from the circular action cache queue, and the action information can be compared with the historical action information. The action information with the larger action amplitude among the historical action information can be updated to the circular action cache queue. This can ensure that the control system maintains strong execution and avoid action degradation caused by model fluctuations.

[0089] In another example, historical action information can be retrieved from the circular action cache queue, and the action information can be weighted and fused with the historical action information. The fused result can then be updated in the circular action cache queue.

[0090] S403. If it does not exist, update the circular action cache queue in the terminal device according to the action information.

[0091] Optionally, if the circular action cache queue does not contain historical action information for the corresponding time point, the action information can be updated in the circular action cache queue.

[0092] In one possible implementation, step S402 above updates the circular action buffer queue in the end-side device based on action information and historical action information, including: The action information and historical action information are weighted and fused to obtain the current action information, which is then filled into the corresponding position in the circular action buffer queue in the edge device.

[0093] Optionally, the action information and historical action information can be weighted and fused to obtain the current action information. The current action information is then filled into the corresponding position of the circular action buffer queue in the end device, which enables a gradual connection between the new and old actions, avoids abrupt changes in control commands, fundamentally solves the trajectory "stuttering" problem, and at the same time, retains some reliable historical action information to prevent overall control instability caused by a single abnormal prediction.

[0094] In one possible implementation, Figure 5 A flowchart illustrating the process of obtaining current action information in the device control method provided in this application embodiment, referring to... Figure 5 As shown, the above steps perform weighted fusion of action information and historical action information to obtain current action information, including: S501. Determine the current scheduling time and the relative time freshness of the action information at the current scheduling time.

[0095] Optionally, the current scheduling time can be determined, and the relative time freshness of the action information at the current scheduling time can be determined.

[0096] Here, the current scheduling time refers to the precise system time corresponding to the current scheduling cycle when the end-side controller is executing the control loop. The relative time freshness of action information at the current scheduling time refers to the "age" of the action information from the current scheduling time. The more recent the action information is, the greater its relative time freshness and the higher its dominance.

[0097] In one example, the action information and the current scheduling time can be input into a pre-trained time freshness determination model, which then predicts the relative time freshness of the action information at the current scheduling time.

[0098] In another example, the action information can be input into a pre-built formula for calculating the relative time freshness of the action information at the current scheduling moment.

[0099] S502, Determine the uncertainty of motion information.

[0100] Optionally, the uncertainty of the action information can be determined based on the uncertainty score of each action information.

[0101] Uncertainty refers to the reliability of action information.

[0102] For example, uncertainty scores for action information can be mapped to confidence weights. Specifically, you can refer to the following formula to achieve this:

[0103] in, For scaling factors (e.g.) ), used to amplify the weights of samples with low uncertainty, Score the uncertainty. , The smaller the value, the more reliable the prediction.

[0104] S503. Based on relative time freshness and uncertainty, the action information and historical action information are weighted and fused to obtain the current action information.

[0105] Optionally, after obtaining the relative time freshness and uncertainty, the action information and historical action information can be weighted and fused according to the relative time freshness and uncertainty to obtain the current action information.

[0106] For example, relative time freshness and uncertainty can be used as weights to perform weighted calculations on action information and historical action information, and then summed to obtain the current action information.

[0107] By determining relative time freshness and uncertainty, and then weighting and fusing action information and historical action information based on these factors, current action information is obtained. This allows the system to perceive contextual changes and automatically determine when to trust new predictions and when to rely on historical experience. Simultaneously, it accurately addresses network latency and communication jitter, ensuring control continuity. Furthermore, it optimizes multi-cycle inference consistency and mitigates model oscillation issues.

[0108] In one possible implementation, Figure 6 This is another schematic flowchart illustrating the process of obtaining current action information in the device control method provided in this application embodiment, referring to... Figure 6 As shown, in S503 above, the action information and historical action information are weighted and fused according to relative time freshness and uncertainty to obtain the current action information, including: S601. Based on relative time freshness and uncertainty, the first weight and the second weight are calculated.

[0109] Optionally, a first weight and a second weight can be calculated based on relative time freshness and uncertainty. The sum of the first weight and the second weight is 1.

[0110] For example, the first weight can be obtained by increasing / decreasing exponentially based on relative time freshness and uncertainty. Specifically, it can be obtained using a time constant or a Bayesian weight.

[0111] For example, the second weight can be obtained by subtracting the first weight from 1.

[0112] S602. Calculate the product of the first weight and the action information to obtain the first product result.

[0113] Optionally, the first weight can be calculated. With motion information The product of these two products yields the first product result. .

[0114] S603. Calculate the product of the second weight and the historical action information to obtain the second product result.

[0115] Optionally, a second weight can be calculated. With historical action information The product of the first and second products yields the second product result. .

[0116] S604. Calculate the sum of the first product result and the second product result, and use it as the current action information.

[0117] Optionally, the first product result can be calculated. The result of the second product The sum of the values ​​serves as the current action information. .

[0118] By considering relative time freshness and uncertainty, a first weight and a second weight are calculated. These weights are then used to calculate the current action information, enabling multi-factor joint decision-making and improving the level of intelligent control. Simultaneously, it automatically suppresses low-quality predictions, preventing the risk of miscontrol. Furthermore, it ensures action continuity and supports seamless transitions under rolling prediction.

[0119] In one possible implementation, Figure 7 A flowchart illustrating the process of determining the current control command in the device control method provided in this application embodiment is shown below. Figure 7 As shown, in S203 above, determining the current control command from the circular action buffer queue based on the current time includes: S701. Based on the current time, determine whether there is action information corresponding to the current time in the circular action buffer queue.

[0120] Optionally, the current time can be used as a query condition. A search algorithm can be used to search the circular action cache queue to determine whether action information corresponding to the current time exists. For example, a binary search algorithm can be used.

[0121] The action information corresponding to the current moment refers to the action information in the circular action cache queue whose timestamp is equal to the current moment.

[0122] S702. If so, generate the current control command based on the action information corresponding to the current moment.

[0123] Optionally, if action information corresponding to the current moment exists, control instructions are generated based on action information in the circular action buffer queue whose timestamps are equal to the current moment.

[0124] S703. If not, then the current control command is generated by interpolation based on the action information corresponding to the previous moment and / or the next moment.

[0125] Optionally, if there is no action information corresponding to the current moment, the action information corresponding to the previous moment and / or the next moment is obtained, and the current control command is generated by linear interpolation or other interpolation algorithms based on the time difference between the current moment and the previous moment and / or the next moment.

[0126] Optionally, if no action information corresponding to the current moment exists, action information corresponding to multiple moments before and / or multiple moments after the current moment can be obtained, and the current control command can be generated by linear interpolation or other interpolation algorithms based on the time difference between the current moment and the multiple moments before and / or multiple moments after.

[0127] By using the "current moment" as the benchmark for querying and interpolation, reasonable control commands can be generated using existing motion information even in the event of lost or delayed prediction data, completely avoiding motion interruptions caused by data loss. Simultaneously, it ensures that the generated motion commands change continuously over time, effectively preventing abrupt changes in velocity or acceleration and eliminating trajectory "stuttering." Furthermore, it significantly reduces reliance on network communication stability, enhancing the overall system's anti-interference capabilities.

[0128] This application also provides an electronic device, such as... Figure 8 As shown, Figure 8 The schematic diagram of the electronic device structure provided in the embodiments of this application includes: a processor 801 and a memory 802, and optionally, a bus 803. The memory 802 stores machine-readable instructions executable by the processor 801. When the electronic device is running, the processor 801 and the memory 802 communicate through the bus 803, and the processor 801 executes the machine-readable instructions to perform the steps of the above-described device control method.

[0129] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the above-described device control method.

[0130] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the method embodiments, and will not be repeated here. In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection can be through some communication interfaces; the indirect coupling or communication connection of devices or modules can be electrical, mechanical, or other forms.

[0131] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. If the functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0132] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A device control method, characterized in that, The method is applied to a device inference system, the device inference system comprising: an edge-side device and multiple end-side devices configured in a local area network, the method comprising: The edge device receives multimodal input data sent by each of the end devices. The multimodal input data includes multimodal sensor data and the end timestamp corresponding to the multimodal sensor data. The edge device performs inference based on the multimodal input data sent by each of the end devices, generates future action prediction information for each of the end devices, and sends the future action prediction information of each of the end devices to each of the end devices. The future action prediction information includes: action information of the end device in multiple future time steps and timestamps corresponding to each action information. Each of the aforementioned end-side devices updates the circular action buffer queue in the end-side device based on the future action prediction information, and determines the current control command from the circular action buffer queue according to the current time, and executes the current control command.

2. The equipment control method according to claim 1, characterized in that, The process of inferring from the multimodal input data sent by each of the terminal devices to generate future action prediction information for each of the terminal devices includes: The global timestamp of the multimodal sensor data is determined based on the end-side timestamp corresponding to the multimodal sensor data. Based on the global timestamps of the multimodal sensor data, determine the multimodal state vectors at multiple time points; Based on the multimodal state vectors at each time point and the pre-trained inference model, the action information of the edge device is generated for multiple time steps in the future. Add corresponding timestamps to the action information of the terminal device at multiple future time steps to obtain the future action prediction information.

3. The equipment control method according to claim 1, characterized in that, The step of updating the circular action cache queue in the terminal device based on the future action prediction information includes: Based on the timestamp corresponding to the action information, determine whether there is historical action information corresponding to the time point in the circular action cache queue in the terminal device; If it exists, the circular action cache queue in the terminal device is updated according to the action information and the historical action information; If it does not exist, the circular action cache queue in the terminal device is updated according to the action information.

4. The equipment control method according to claim 3, characterized in that, The step of updating the circular action cache queue in the terminal device based on the action information and the historical action information includes: The action information and the historical action information are weighted and fused to obtain the current action information, and the current action information is filled into the corresponding position of the circular action cache queue in the terminal device.

5. The equipment control method according to claim 4, characterized in that, The weighted fusion of the action information and the historical action information to obtain the current action information includes: Determine the current scheduling time and the relative time freshness of the action information at the current scheduling time; The uncertainty of the action information is determined; Based on the relative time freshness and the uncertainty, the action information and the historical action information are weighted and fused to obtain the current action information.

6. The equipment control method according to claim 5, characterized in that, The step of weighting and fusing the action information and the historical action information based on the relative time freshness and the uncertainty to obtain the current action information includes: Based on the relative time freshness and the uncertainty, the first weight and the second weight are calculated; Calculate the product of the first weight and the action information to obtain the first product result; Calculate the product of the second weight and the historical action information to obtain the second product result; The sum of the first product result and the second product result is calculated and used as the current action information.

7. The equipment control method according to claim 1, characterized in that, The step of determining the current control command from the circular action buffer queue based on the current time includes: Based on the current time, determine whether there is action information corresponding to the current time in the circular action buffer queue; If so, then generate the current control command based on the action information corresponding to the current moment; If not, the current control command is generated by interpolation based on the action information corresponding to the previous moment and / or the next moment.

8. A device inference system, characterized in that, The device inference system includes: an edge-side device and multiple end-side devices located in a local area network, and the device inference system is used to execute the device control method according to any one of claims 1-7.

9. An electronic device, characterized in that, include: A processor and a memory, the memory storing machine-readable instructions executable by the processor, wherein when the electronic device is in operation, the processor executes the machine-readable instructions to perform the device control method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, performs the device control method as described in any one of claims 1-7.

Images