A large model end side reasoning acceleration method of embodied intelligent industrial robots

Abstract

The application discloses a large model end side reasoning acceleration method of a body intelligent industrial robot, relates to the technical field of large models, and collects visual feature images and end torque signals, splices visual feature labels of target process mapping features, global task labels and additional two-dimensional metadata, and generates a multimodal input feature stream. When determining process transition, a three-dimensional reachable envelope is solved, image feature tensors that are separated from the envelope by inverse projection are divided into redundant transition tensors, sequence substantial compression and position index continuous remapping are performed when updating a key-value cache logic block mapping table, global process anchor point tensors are asynchronously transported to on-chip static random access memory and anchored to the starting absolute position index of the generated multimodal input feature stream, feature stream generation queries and local key-value tensors are used to calculate attention by splicing with anchors, target poses and compliant parameters are decoded, and the torque control sequence drives the motor by inverse dynamics calculation.

Description

Technical Field

[0001] This invention relates to the field of large model technology, and in particular to a method for accelerating end-side inference of large models of embodied intelligent industrial robots. Background Technology

[0002] In recent years, the combination of embodied intelligent industrial robots and multimodal large language models has become a research hotspot in the field of intelligent manufacturing. However, when multimodal large language models are deployed in industrial edge computing units, the ultra-long context sequences and high-resolution visual inputs can lead to severe physical memory overflow and computing power bottlenecks.

[0003] Currently, Chinese invention patent CN121562791A discloses a method and system for accelerating inference in multimodal large models on the edge. By performing two-stage screening and rearranging of visual tokens based on CLS attention scores and cross-modal attention from text to vision in the visual encoder and pre-filling stages, a sparse key-value cache is constructed. In the decoding stage, important neuron sets are determined based on activation gating or historical statistics. Feedforward network weights are retrieved on demand through hierarchical storage and asynchronous I / O, thereby reducing memory usage and computational overhead. The aforementioned technology, based on a purely data-driven selection mechanism using attention scores within the model, suffers from fundamental physical logic flaws and implementation risks when directly applied to embodied intelligent industrial robots. First, the visual token selection mechanism relies entirely on semantic similarity or attention probability distribution at the two-dimensional image level, detached from the geometric constraints of three-dimensional objective physical space. When embodied intelligent robots perform physical processes, the end effector, constrained by the kinematic laws and physical limit parameters of the underlying servo motors, can only interact with the environment within an extremely limited three-dimensional reachable envelope. Tensor pruning at the purely semantic probability level is highly susceptible to mistakenly deleting key visual features that are on the edge of physical collision but whose attention scores have not yet become prominent. Blindly retaining a large number of redundant background features in the video memory, which are semantically related but physically unreachable in the current process, results in a huge waste of computing power in the edge video memory. Second, the lack of a deterministic mapping mechanism between the underlying physical entity constraints and the large model cache management means that for industrial-grade closed-loop control tasks, if the release of tensor cache is based solely on statistical thresholds without objective physical laws as hard boundary judgment criteria, the virtual cognitive boundary of the multimodal large language model will be torn apart from the underlying electromechanical control entity space. This probabilistic cache erasure cannot guarantee the absolute security required by industrial control and is difficult to meet the need for strict physical locking of the context attention span in cross-process physical operations. Summary of the Invention

[0004] The technical problem solved by this invention is that existing embodied intelligent industrial robots face problems such as physical memory overflow, dynamic obstacle avoidance risks, and virtual features deviating from the boundaries of objective physical reality in long-sequence multimodal large language model edge inference, which can easily lead to accidental deletion of process references and ultimately cause physical collisions and assembly failures of the robot.

[0005] To address the aforementioned technical problems, this invention provides the following technical solution: a method for accelerating end-to-end inference of a large-scale embodied intelligent industrial robot, comprising the following steps: Step S1: Extract the target process identifier, collect visual feature images and end torque signals, and concatenate the mapping features of the target process identifier and end torque signals, global task markers, and visual feature markers of additional two-dimensional visual block coordinate sideband metadata to generate a multimodal input feature stream. Step S2: Extract the target process identifier of the current time step and the historical process identifier of the previous time step. When the target process identifier and the historical process identifier are inconsistent, determine that a process jump event has occurred. Locate the logical index corresponding to the global task marker in the input sequence of the previous process. Extract the key value tensor in the corresponding physical memory block in the paging key value tensor cache pool and divide the key value tensor into global process anchor point tensors. Solve the three-dimensional reachable envelope. Obtain the image feature tensor in the paging key value tensor cache pool that belongs to the previous process and corresponds to the visual feature marker. Read the two-dimensional visual block coordinate sideband metadata that is synchronously bound to the image feature tensor. Based on the two-dimensional visual block coordinate sideband metadata, back-project the image feature tensor into a three-dimensional scene point cloud. Divide the image feature tensor that is spatially detached from the three-dimensional reachable envelope into redundant transition tensors. Step S3: Release the physical memory block occupied by the redundant transition tensor. When updating the key-value cache logical block mapping table, delete the logical index entry corresponding to the redundant transition tensor from the key-value cache logical block mapping table to perform substantial sequence compression. For the surviving image feature tensor that has not been released, based on the continuous sequential index of the surviving image feature tensor in the compressed sequence, call the position encoding remapping operator to recalculate the rotation position encoding matrix of the surviving image feature tensor to perform continuous position index remapping. Asynchronously move the global process anchor tensor to the on-chip static random access memory through the direct memory access channel, and anchor the global process anchor tensor to the starting absolute position index of the generated multimodal input feature stream. Step S4: Generate a feature query vector sequence and a local key value tensor based on the multimodal input feature stream. Read the locked global process anchor point tensor and concatenate the local key value tensor to calculate the attention weight. Output the hidden layer state vector. Decode and generate the target pose sequence and compliant impedance parameters and send them to the inverse dynamics solver to solve for the joint torque control sequence to drive the motor.

[0006] Preferably, step S1 includes the following sub-steps: Step S101: Read the process state machine instructions issued in real time by the underlying programmable logic controller, decode the process state machine instructions, and extract the target process identifier that needs to be executed. Step S102: At the current time step, capture a visual feature image at a first sampling frequency and assign a first hardware timestamp, and at the same time capture an end torque signal at a second sampling frequency and record the corresponding second hardware timestamp sequence. The second sampling frequency is greater than the first sampling frequency. Step S103: Using the first hardware timestamp as the cutoff time, trace back to the sampling period corresponding to one of the first sampling frequencies to construct a causal sliding time window; Extract the end torque signal that falls within the causal sliding time window with the second hardware timestamp, perform feature aggregation on the extracted end torque signal, and calculate the torque peak value, torque mean value and torque temporal variance within the causal sliding time window respectively. Based on the feature aggregation results, generate the downsampled aligned torque feature vector.

[0007] Preferably, step S1 further includes the following sub-steps: Step S104: Map the target process identifier and the alignment torque feature vector through a preset linear projection layer respectively; The output dimension of the linear projection layer is constrained to be consistent with the feature dimension of the visual feature image output by the visual encoder, so as to obtain the process condition mark and force feature mark of the first dimension. The preset global task markers, process condition markers, force feature markers, and visual feature markers with additional two-dimensional visual block coordinate sideband metadata are extracted, concatenated along the sequence length dimension, and fused to generate a multimodal input feature stream suitable for attention mechanism computation.

[0008] Preferably, step S2 includes the following sub-steps: Step S201: In the forward inference process of the multimodal large language model, extract the target process identifier of the current time step and the historical process identifier of the previous time step, compare the target process identifier with the historical process identifier, and determine that a process jump event has occurred when the target process identifier and the historical process identifier are inconsistent. Step S202: In response to the process transition event, query the key-value cache logical block mapping table, locate the logical index corresponding to the global task marker in the input sequence of the previous process, extract the key-value tensor in the corresponding physical memory block in the paging key-value tensor cache pool based on the located logical index, and divide the key-value tensor into global process anchor point tensors.

[0009] Preferably, step S2 further includes the following sub-steps: Step S203: Query the preset industrial standard operating procedure knowledge graph, extract the motion limit parameters of each joint of the end effector that are mapped to the target process identifier, substitute each joint motion limit parameter into the forward kinematics model, and calculate the three-dimensional reachable envelope of the end effector under the target process through the boundary search algorithm.

[0010] Preferably, step S2 further includes the following sub-steps: Step S204: Obtain the image feature tensor belonging to the previous process in the pagination key value tensor cache pool, read the two-dimensional visual block coordinate sideband metadata that is synchronously bound to the image feature tensor, extract the pixel-level depth matrix of the corresponding image block from the depth information of the visual sensor based on the two-dimensional visual block coordinate sideband metadata, and perform coordinate system inverse transformation on the pixel-level depth matrix using the camera intrinsic and extrinsic parameter matrix to generate a three-dimensional scene point cloud located in the robot base coordinate system. Step S205: Perform pixel-level spatial difference operation on the point cloud of the three-dimensional scene in two consecutive frames to isolate the dynamic feature region where the spatial depth change is greater than a preset threshold. For discrete points within the dynamic feature region, the instantaneous three-dimensional velocity vector is calculated using the first-order forward difference method; By introducing a look-ahead window parameter, and based on the instantaneous three-dimensional velocity vector and the look-ahead window parameter, a linear manifold projection is performed on the discrete points within the dynamic feature region to obtain the projected trajectory manifold; At each time sampling point of the projected trajectory manifold, the three-dimensional spatial Euclidean distance between each time sampling point and the outer boundary of the three-dimensional reachable envelope is calculated using the signed distance field function, and the minimum value at all time sampling points is taken as the shortest safe distance. When it is determined that the 3D scene point cloud corresponding to the preset image feature tensor is completely out of the 3D reachable envelope at the current time step, and the shortest safe distance corresponding to the preset image feature tensor is greater than the preset safety tolerance threshold, the preset image feature tensor is divided into a redundant transition tensor.

[0011] Preferably, step S3 includes the following sub-steps: Step S301: In response to the classification results of the global process anchor tensor and the redundant transition tensor, the paging memory management interface is called through the edge computing unit to obtain the physical memory addressing base address and physical memory data block length of the redundant transition tensor mapped in the paging key-value tensor cache pool. Step S302: A release command is sent to the video memory controller through the paging memory management interface. The video memory controller releases the logical address binding of the redundant transition tensor in the key-value cache logical block mapping table according to the physical video memory addressing base address and the physical video memory data block length, and forcibly releases the physical video memory block occupied by the redundant transition tensor. When updating the key-value cache logical block mapping table, the logical index entries corresponding to redundant transition tensors are deleted from the key-value cache logical block mapping table. For surviving image feature tensors that have not been released, the rotation position encoding matrix of the surviving image feature tensors is recalculated by calling the position encoding remapping operator based on the continuous sequential index of the surviving image feature tensors in the compressed sequence.

[0012] Preferably, step S3 further includes the following sub-steps: Step S303: While issuing the release instruction, configure the source address of the direct memory access channel as the first storage address of the global process anchor tensor in the paging key-value tensor cache pool, and configure the destination address of the direct memory access channel as the second storage address in the on-chip static random access memory. Start the direct memory access channel to asynchronously move the global process anchor tensor from the first storage address to the second storage address. After the transfer is completed, when generating the multimodal input feature stream of the next target process, the global process anchor tensor is inserted as the first feature marker of the input sequence, and the absolute position index 0 is assigned to the global process anchor tensor. The subsequent real-time feature markers are then sequentially concatenated from the absolute position index 1.

[0013] Preferably, step S4 includes the following sub-steps: Step S401: In the subsequent reasoning stage of the target process, extract the multimodal input feature stream of real-time input, and generate the feature query vector sequence and local key-value tensor of the current time step through linear transformation; Step S402: Read the global process anchor tensor locked in the on-chip static random access memory through the internal data bus, concatenate the global process anchor tensor and the local key value tensor in the sequence length dimension, calculate the similarity matrix with the feature query vector sequence, perform attention weight allocation, and output the hidden layer state vector.

[0014] Preferably, step S4 includes the following sub-steps: Step S403: Input the hidden layer state vector into a preset motion decoding network for mapping and decoding to generate the target pose sequence and compliant impedance parameters of the end effector; Step S404: The target pose sequence and compliant impedance parameters are sent to the inverse dynamics solver. The inverse dynamics solver performs dynamic calculations based on a preset servo control cycle and converts them into a joint torque control sequence to drive the servo motor to perform physical actions.

[0015] The beneficial effects of this invention are as follows: First, it introduces precise memory reclamation based on time-varying prediction. When determining the occurrence of process transition events, it breaks through the limitations of traditional static collision detection. Based on the time-varying prediction mechanism of servo look-ahead window and symbolic distance field, combined with the instantaneous velocity manifold projection of three-dimensional reachable envelope and discrete points, it accurately divides image features that are both temporally and spatially free from physical interaction risks into redundant transition tensors. Under the premise of providing dynamic safety redundancy for the robot, it eliminates the waste of end-side computing power caused by background features in non-working areas. Second, after forcibly releasing redundant memory blocks, it abandons the traditional hole stitching method that destroys position encoding and executes a substantial sequence compression and position index continuous remapping mechanism, which achieves the maximum physical memory utilization. During the recovery process, the remapping operator is invoked to recalibrate the rotation position encoding of the survival features, maintaining the temporal topological order and causal validity of the autoregressive inference of the large model. Finally, after asynchronously transferring the global process anchor tensor to the on-chip static random access memory through the direct memory access channel, the zero-index absolute anchoring mechanism is used to force the global process anchor tensor to be inserted at the beginning of the next process sequence. This conforms to and stimulates the natural attention sedimentation effect of the pre-trained large model. Without tampering with the relative position bias parameter matrix of the original model, stable hard locking of cross-process task semantics is achieved, ensuring that the instructions output by the large model can be transformed into a precise joint torque control sequence by the inverse dynamics solver. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating the steps of a method for accelerating end-to-end inference of a large-scale embodied intelligent industrial robot, as provided in one embodiment of the present invention. Detailed Implementation

[0017] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0018] Example, refer to Figure 1 This paper provides a method for accelerating edge-side inference of large-scale embodied intelligent industrial robots, including the following steps: Step S1: Extract the target process identifier, collect visual feature images and end torque signals, and concatenate the mapping features of the target process identifier and end torque signals, global task markers, and visual feature markers of the additional two-dimensional visual block coordinate sideband metadata to generate a multimodal input feature stream. Step S2: Extract the target process identifier of the current time step and the historical process identifier of the previous time step. When the target process identifier and the historical process identifier are inconsistent, determine that a process jump event has occurred. Locate the logical index corresponding to the global task marker in the input sequence of the previous process. Extract the key-value tensor in the corresponding physical memory block in the paging key-value tensor cache pool and divide the key-value tensor into global process anchor point tensors. Solve the three-dimensional reachable envelope. Obtain the image feature tensor in the paging key-value tensor cache pool that belongs to the previous process and corresponds to the visual feature marker. Read the two-dimensional visual block coordinate sideband metadata that is synchronously bound to the image feature tensor. Based on the two-dimensional visual block coordinate sideband metadata, back-project the image feature tensor into a three-dimensional scene point cloud. Divide the image feature tensor that is spatially detached from the three-dimensional reachable envelope into redundant transition tensors. Step S3: Release the physical memory block occupied by the redundant transition tensor. When updating the key-value cache logical block mapping table, delete the logical index entry corresponding to the redundant transition tensor from the key-value cache logical block mapping table to perform substantial sequence compression. For the surviving image feature tensor that has not been released, based on the continuous sequential index of the surviving image feature tensor in the compressed sequence, call the position encoding remapping operator to recalculate the rotation position encoding matrix of the surviving image feature tensor to perform continuous position index remapping. Asynchronously move the global process anchor tensor to the on-chip static random access memory through the direct memory access channel, and anchor the global process anchor tensor to the starting absolute position index of the generated multimodal input feature stream. Step S4: Generate a feature query vector sequence and a local key value tensor based on the multimodal input feature flow. Read the locked global process anchor point tensor and concatenate the local key value tensor to calculate the attention weight. Output the hidden layer state vector. Decode and generate the target pose sequence and compliant impedance parameters and send them to the inverse dynamics solver to solve for the joint torque control sequence to drive the motor.

[0019] This invention first introduces time-varying prediction for precise memory reclamation. When determining the occurrence of process transition events, it overcomes the limitations of traditional static collision detection. Based on a time-varying prediction mechanism using a servo look-ahead window and symbolic distance field, combined with the instantaneous velocity manifold projection of the 3D reachable envelope and discrete points, it accurately divides image features that are both temporally and spatially free from physical interaction risks into redundant transition tensors. This provides dynamic safety redundancy for the robot while eliminating the waste of end-side computing power caused by background features in non-working areas. Secondly, after forcibly releasing redundant memory blocks, it abandons the traditional hole-stitching method that destroys position encoding, and executes a substantial sequence compression and position index continuous remapping mechanism, achieving maximum physical memory reclamation. Simultaneously, the rotation position encoding of the survival feature is recalibrated by calling the remapping operator, maintaining the temporal topological order and causal validity of the autoregressive inference of the large model. Finally, after the global process anchor tensor is asynchronously transported to the on-chip static random access memory through the direct memory access channel, the zero-index absolute anchoring mechanism is used to force the global process anchor tensor to be inserted at the beginning of the next process sequence. This conforms to and stimulates the natural attention sedimentation effect of the pre-trained large model. Without tampering with the relative position bias parameter matrix of the original model, stable hard locking of cross-process task semantics is achieved, ensuring that the instructions output by the large model can be transformed into a precise joint torque control sequence by the inverse dynamics solver.

[0020] Step S1 includes the following sub-steps: Step S101: Read the process state machine instructions issued in real time by the underlying programmable logic controller, decode the process state machine instructions, and extract the target process identifier that needs to be executed.

[0021] The underlying programmable logic controller (PLC), serving as the central hardware control unit for the embodied intelligent industrial robot, manages the underlying states of the servo motors. The process state machine instructions include information about the robot's current macroscopic work stage, encompassing instruction codes for compliant grasping, precision assembly, or surface polishing. By decoding these instructions, the system can extract a unique target process identifier. This identifier provides explicit contextual prior knowledge for the subsequent multimodal large language model, enabling it to adaptively adjust attention weight allocation strategies based on the current processing stage.

[0022] Step S102: At the current time step, capture a visual feature image at a first sampling frequency and assign a first hardware timestamp, and at the same time capture an end torque signal at a second sampling frequency and record the corresponding second hardware timestamp sequence. The second sampling frequency is greater than the first sampling frequency.

[0023] Due to the significant differences in the physical characteristics of different modal sensors, data acquisition must be decoupled. At the current time step, the vision sensor mounted on the embodied intelligent industrial robot captures visual feature images at a first sampling frequency, and the underlying hardware clock source assigns a first hardware timestamp to the visual feature images. The first sampling frequency is set to a standard 30 Hz. Simultaneously, the six-dimensional torque sensor mounted on the end effector of the embodied intelligent industrial robot continuously captures the end effector torque signal at a second sampling frequency and records a second hardware timestamp sequence corresponding to each sample. The second sampling frequency is set to a high-frequency 1000 Hz. Setting the second sampling frequency higher than the first sampling frequency ensures that within the low-frequency period of the vision sensor generating one frame of image data, the system can acquire sufficiently dense high-frequency torque change data, thus avoiding missing instantaneous physical contact collision signals.

[0024] Step S103: Using the first hardware timestamp as the cutoff time, trace back to the sampling period corresponding to one of the first sampling frequencies to construct a causal sliding time window.

[0025] Extract the end torque signal that falls within the causal sliding time window of the second hardware timestamp, perform feature aggregation on the extracted end torque signal, and calculate the torque peak value, torque mean value and torque temporal variance within the causal sliding time window respectively. Based on the feature aggregation results, generate the downsampled aligned torque feature vector.

[0026] Step S103 is the core of aligning the high-frequency torque signals with the low-frequency visual signals in terms of timing. To strictly adhere to the causal rules of the model inference and avoid future data leakage, the system must use the first hardware timestamp as the absolute cutoff point and trace back to a time period exactly equal to one visual sampling cycle, forming a causal sliding time window. All high-frequency end-torque signals whose hardware timestamps fall within the causal sliding time window are extracted. To transform these high-frequency one-dimensional signals into high-dimensional features that can be processed by a large language model, the extracted end-torque signals undergo dimensionality reduction and feature aggregation: Calculate the peak torque to characterize the maximum physical impact force occurring within the causal sliding time window.

[0027] The mean torque is calculated to characterize the overall force trend within the causal sliding time window.

[0028] The torque time series variance is calculated to characterize the vibration frequency characteristics and force stability of the end effector within the causal sliding time window.

[0029] The peak torque, mean torque, and temporal variance of torque are concatenated to generate an aligned torque feature vector that is perfectly aligned with the visual feature image on the time axis.

[0030] Step S104: Map the target process identifier and the alignment torque feature vector through a preset linear projection layer.

[0031] The output dimension of the constrained linear projection layer is consistent with the feature dimension of the visual feature image output by the visual encoder, thus obtaining the first dimension of process condition label and force feature label.

[0032] The preset global task markers, process condition markers, force feature markers, and visual feature markers with additional two-dimensional visual block coordinate sideband metadata are extracted, concatenated along the sequence length dimension, and fused to generate a multimodal input feature stream suitable for attention mechanism computation.

[0033] To ensure that data from different modalities can be uniformly input into the Transformer architecture of a multimodal large language model for self-attention matrix multiplication and feature space dimensionality alignment, the target process identifier and alignment moment feature vector are respectively input into a pre-defined linear projection layer for dimensionality upscaling. During the mapping process, the output dimension of the linear projection layer is constrained to be equal to the embedding dimension of the visual feature labels output after the visual feature image has been processed by the visual encoder.

[0034] After dimensional alignment, process condition markers and force feature markers with the first dimension are obtained. Preset global task markers are acquired. Global task markers are used to macroscopically guide the generation of special instructions by the model. The global task markers, process condition markers, force feature markers, and visual feature markers are then concatenated along the sequence length dimension.

[0035] It is worth emphasizing that the visual feature labels are appended with two-dimensional visual block coordinate sideband metadata. This metadata records the absolute two-dimensional pixel coordinates of each visual feature label in the original visual feature image. Completely preserving the two-dimensional absolute coordinate information is a necessary engineering prerequisite to ensure that the abstract feature tensor can be accurately inversely projected back to the physical three-dimensional point cloud. Through the concatenation operation of all labels, a multimodal input feature stream with a standardized format and complete spatiotemporal and mechanical physical properties is finally generated.

[0036] Step S2 includes the following sub-steps: Step S201: In the forward inference process of the multimodal large language model, extract the target process identifier of the current time step and the historical process identifier of the previous time step, compare the target process identifier with the historical process identifier, and determine that a process transition event has occurred when the target process identifier and the historical process identifier are inconsistent.

[0037] In step S202, in response to the process transition event, the key-value cache logical block mapping table is queried, the logical index corresponding to the global task mark in the input sequence of the previous process is located, and based on the located logical index, the key-value tensor in the corresponding physical memory block in the paging key-value tensor cache pool is extracted, and the key-value tensor is divided into global process anchor point tensors.

[0038] It should be noted that, since the global task tag continuously aggregates the contextual semantics of process condition tags, force feature tags, and visual feature tags within the same sequence in the multi-layer self-attention forward propagation process of the multimodal large language model, the corresponding key-value tensor has implicitly encoded the complete process state summary of the previous process. Therefore, using the key-value tensor of the global task tag as a cross-process semantic anchor has sufficient information.

[0039] Step S203: Query the preset industrial standard operating procedure knowledge graph, extract the motion limit parameters of each joint of the end effector that are mapped to the target process identifier, substitute each joint motion limit parameter into the forward kinematics model, and calculate the three-dimensional reachable envelope of the end effector under the target process through the boundary search algorithm.

[0040] In the specific implementation process, a pre-built industrial standard operating procedure knowledge graph pre-constructs a dictionary linking the physical control layer and the semantic logic layer. Specifically, the pre-built industrial standard operating procedure knowledge graph uses different process identifiers as entity nodes and pre-associates them with the robot's kinematic parameter library. When a specific target process identifier is received, the motion limit parameters of each joint under the constraint of that process can be directly retrieved through the graph index, ensuring that the process state machine instructions can be directly converted into underlying physical boundary conditions. The motion limit parameters of each joint include static geometric limit parameters and dynamic dynamic limit parameters. The static geometric limit parameters include the limit safe rotation angle range of each rotary joint that defines the absolute physical workspace boundary of the robot, and the limit effective linear travel range of each translational joint. The dynamic dynamic limit parameters include the maximum physical angular velocity limit and the maximum physical angular acceleration limit of each joint under the output power constraint of the underlying servo driver. The dynamic dynamic limit parameters are used to combine the temporal span of time window resampling to converge the three-dimensional reachable envelope to the true physically reachable subset of the end effector in the current time step.

[0041] Step S204: Obtain the image feature tensor belonging to the previous process from the paging key value tensor cache pool, read the two-dimensional visual block coordinate sideband metadata that is synchronously bound to the image feature tensor, extract the pixel-level depth matrix of the corresponding image block from the depth information of the visual sensor based on the two-dimensional visual block coordinate sideband metadata, and perform coordinate system inverse transformation on the pixel-level depth matrix using the camera intrinsic and extrinsic parameter matrices to generate a three-dimensional scene point cloud located in the robot base coordinate system.

[0042] Step S205 introduces a time-varying prediction mechanism based on spatiotemporal manifold projection and symbolic distance field to calculate the 3D spatial relationship between the 3D coordinates of each discrete point in the 3D scene point cloud and the outer boundary of the 3D reachable envelope. Specifically: Pixel-level spatial difference operation is performed on the point cloud of a 3D scene in two consecutive frames to isolate dynamic feature regions where the spatial depth change is greater than a preset threshold. For discrete points within a dynamic feature region, the instantaneous three-dimensional velocity vector is calculated using the first-order forward difference method. A look-ahead window parameter, defined by the total delay of the system's worst-case response, is introduced. Based on the instantaneous 3D velocity vector and the look-ahead window parameter, a linear manifold projection is performed on the spatial trajectory of discrete points within the dynamic feature region over the future look-ahead period, resulting in the projected trajectory manifold. Even if dynamic objects in the scene continue to move within the robot's perception, reasoning, communication, and braking response periods, the projected trajectory of the dynamic objects will not affect the 3D reachability envelope. The look-ahead window parameter satisfies the following condition: ; in, For look-ahead window parameters, This refers to the mechanical braking safety redundancy time for the end effector of an industrial robot from triggering an emergency stop command to the complete stop of the servo motor. The forward propagation time for end-to-end inference in a multimodal large language model. This refers to the data bus communication delay.

[0043] At each time sampling point of the projected trajectory manifold, the three-dimensional spatial Euclidean distance between each time sampling point and the outer boundary of the three-dimensional reachable envelope is calculated using the signed distance field function, and the minimum value at all time sampling points is taken as the shortest safe distance. When it is determined that the 3D scene point cloud corresponding to the preset image feature tensor is completely out of the 3D reachable envelope at the current time step, and the shortest safe distance corresponding to the preset image feature tensor is greater than the preset safety tolerance threshold, the preset image feature tensor is divided into a redundant transition tensor.

[0044] The preset safety tolerance threshold is dynamically converged and calibrated based on the positioning repeatability of the end effector of the industrial robot and the standard deviation of the depth noise of the vision sensor. The value range of the safety tolerance threshold is 1mm to 10mm.

[0045] An exclusion mechanism is adopted, which requires all discrete points inside the 3D scene point cloud to be detached from the 3D reachable envelope and the projection trajectory manifold to meet the safety distance. This provides safety redundancy for the physical interaction and dynamic obstacle avoidance of industrial robots and prevents the accidental deletion of edge visual features that are in a critical state of physical collision at the edge.

[0046] Step S3 includes the following sub-steps: In step S301, in response to the classification results of the global process anchor tensor and the redundant transition tensor, the paging memory management interface is called through the edge computing unit to obtain the physical memory addressing base address and physical memory data block length mapped by the redundant transition tensor in the paging key-value tensor cache pool.

[0047] The multimodal large language model of embodied intelligent industrial robots typically employs a non-contiguous paging memory architecture to manage the context cache. In response to the classification results, the absolute physical address information of the underlying hardware is required to perform erasure. The edge computing unit calls the underlying paging memory management interface to query the system's virtual memory mapping table, accurately obtaining the physical memory addressing base address where the redundant transition tensor is actually stored in the paging key-value tensor cache pool, as well as the corresponding physical memory data block length. Obtaining the physical memory addressing base address and the physical memory data block length is necessary data preparation for subsequently issuing precise hardware-level reclamation instructions to the memory controller.

[0048] In step S302, a release command is sent to the video memory controller through the paging memory management interface. The video memory controller, based on the physical video memory addressing base address and the physical video memory data block length, unbinds the logical address corresponding to the redundant transition tensor in the key-value cache logical block mapping table, and forcibly releases the physical video memory block occupied by the redundant transition tensor.

[0049] When updating the key-value cache logical block mapping table, the logical index entries corresponding to redundant transition tensors are deleted from the key-value cache logical block mapping table to shorten the total length of the logical sequence. For surviving image feature tensors that have not been released, the position encoding remapping operator is called to recalculate the rotation position encoding matrix of the surviving image feature tensors based on the continuous order index of the surviving image feature tensors in the compressed sequence, so as to maintain the temporal topological order relationship between the surviving image feature tensors.

[0050] When performing the positional encoding remapping operator, the spatiotemporal weights of the live image feature tensor are reconstructed using the following mathematical model: For the logical index in the compressed sequence is Extract the original physical index corresponding to the survival image feature tensor before performing redundancy release. ; Based on the original physical index Calculate the rotation angle after remapping Rotation angle The mathematical expression is: ; in, This represents the absolute position index of the survival image feature tensor within the initial multimodal input feature stream. The rotational base angle constants are determined during the pre-training phase of the multimodal large language model.

[0051] By using the original physical index Re-inject current compressed sequence position During the rotation matrix calculation, the relative rotation phase difference of the surviving image feature tensors participating in the self-attention calculation remains constant in the complex feature space. This remapping mechanism ensures that even after physical-level continuous compression is performed on the key-value cache, the relative topological distance between tensors perceived by the model remains logically consistent with that before redundancy release, thereby eliminating inference offset caused by index hole stitching while maximizing memory reclamation.

[0052] In one embodiment, a release command is sent to the underlying video memory controller via the paging memory management interface. Upon receiving the release command, the video memory controller, based on the physical video memory addressing base address and the physical video memory data block length, directly unbinds the virtual logical address corresponding to the redundant transition tensor in the key-value cache logical block mapping table, marks the physical video memory block previously occupied by the redundant transition tensor as available, and forcibly reclaims it to the system's video memory free pool.

[0053] When updating the key-value cache logical block mapping table, a mechanism for substantial sequence compression and position index continuum remapping is implemented, specifically as follows: The logical index entries corresponding to redundant transition tensors are removed from the key-value cache logical block mapping table, shortening the total length of the logical sequence and achieving substantial physical compression of the sequence length. For surviving image feature tensors that have not been released, the rotation position encoding matrix of the surviving image feature tensors is recalculated by calling the position encoding remapping operator based on the continuous sequential index of the surviving image feature tensors in the compressed sequence.

[0054] It should be noted that the continuous remapping operation, while maintaining the temporal and topological order of the surviving image feature tensors, compresses the relative positional distances that were originally separated by redundant transition tensors. Since the attention decay function of the rotated position encoding monotonically decreases with relative distance, the attention coupling strength between the surviving tensors will be slightly enhanced after compression. However, the image features corresponding to the deleted redundant transition tensors are determined to be irrelevant to the current task in the three-dimensional physical space and will not participate in subsequent self-attention calculations. The previously occupied intermediate position indices no longer contribute information to the semantic associations between surviving tensors; therefore, the attention score shift caused by relative distance compression is within the acceptable causal error range for engineering applications. The remapped position encoding can maintain the causal validity of large-scale autoregressive inference while maximizing the utilization of physical memory.

[0055] In step S303, while issuing the release instruction, the source address of the direct memory access channel is configured as the first storage address of the global process anchor tensor in the paging key-value tensor cache pool, and the destination address of the direct memory access channel is configured as the second storage address in the on-chip static random access memory. The direct memory access channel is then started to asynchronously move the global process anchor tensor from the first storage address to the second storage address.

[0056] After the transfer is completed, when generating the multimodal input feature stream of the next target process, the global process anchor tensor is inserted as the first feature label of the input sequence and the absolute position index 0 is fixed. The real-time feature labels sampled subsequently are sequentially concatenated from the absolute position index 1. The natural attention sedimentation effect of the large language model is used to make the global process anchor tensor continuously attract attention weights, thereby realizing semantic locking across processes.

[0057] In one embodiment, before issuing the release command, a page-locking flag is set on the physical memory block where the global process anchor tensor resides to prevent it from being mistakenly reclaimed or migrated by the memory controller during the release phase. Then, simultaneously with issuing the release command, the source address of the direct memory access channel is configured as the first storage address of the global process anchor tensor in the page key-value tensor cache pool, and the destination address of the direct memory access channel is configured as the second storage address in the on-chip static random access memory. The direct memory access channel is then started to asynchronously move the global process anchor tensor from the first storage address to the second storage address.

[0058] After the transfer and solidification are completed, a zero-index absolute anchoring mechanism is adopted to adapt to the native attention distribution rules of multimodal large language models, specifically: When generating the multimodal input feature stream for the next target process, the global process anchor tensor is forced to be inserted as the first feature label of the input sequence, and the absolute position index 0 is constantly and fixedly assigned to the global process anchor tensor.

[0059] The real-time visual feature markers, force feature markers, and task markers that are subsequently sampled are sequentially spliced ​​from absolute position index 1, maintaining the relative positional relationship of the real-time visual feature markers and force feature markers in the subsequent time sequence.

[0060] By leveraging the attention sedimentation effect naturally formed by sequence start markers during the pre-training stage of the multimodal large language model, the global process anchor tensor continuously absorbs a high amount of self-attention weight distribution during the long-term autoregressive inference stage across processes. Without altering the relative position bias parameter matrix of the original model, it perfectly performs hard locking of the underlying semantics across processes, avoiding the model from generating process illusions.

[0061] Step S4 includes the following sub-steps: Step S401: In the subsequent reasoning stage of the target process, extract the multimodal input feature stream of real-time input, and generate the feature query vector sequence and local key value tensor of the current time step through linear transformation.

[0062] During the subsequent inference stage of autoregressive generation in the multimodal large language model, the system continuously extracts the multimodal input feature stream input in real time at the current time step. The multimodal input feature stream is then fed into the query weight matrix, key weight matrix, and value weight matrix of the multimodal large language model, respectively, and a linear transformation projection is performed on the high-dimensional feature space. After the linear transformation projection, the multimodal input feature stream is deconstructed and mapped into a sequence of feature query vectors and a local key-value tensor corresponding to the current time step. The feature query vector sequence represents the real-time query interaction request in the current physical environment, and the local key-value tensor encodes the local physical context information of the current transient.

[0063] Step S402: Read the global process anchor tensor locked in the on-chip static random access memory through the internal data bus, concatenate the global process anchor tensor and the local key value tensor in the sequence length dimension, calculate the similarity matrix with the feature query vector sequence, perform attention weight allocation, and output the hidden layer state vector.

[0064] To integrate cross-process global task memory with extremely low physical latency, a high-bandwidth internal data bus directly reads the hard-locked global process anchor tensor from the on-chip static random access memory in a zero-copy manner. Subsequently, the system concatenates the global process anchor tensor representing macroscopic task semantics with the local key-value tensor representing transient physical information along the tensor's sequence length dimension, constructing a complete global and local composite key-value tensor. The self-attention computation engine of the multimodal large language model performs a dot product operation on the feature query vector sequence and the composite key-value tensor, and performs division scaling using the square root of the key vector dimension as a scaling factor to calculate a similarity matrix reflecting the correlation of the feature space. After Softmax normalization and attention weight allocation, the system weights the allocated attention weights and finally outputs the hidden layer state vector representing the multimodal decision instructions, the mathematical expression of which is: ; in, For feature query vector sequence With composite key tensor Perform dot product operations to calculate the original similarity scores between different labels in the feature space. The square root of the dimension of the key vector. This serves as a scaling factor to prevent the dot product result from becoming too large and entering the saturation region of Softmax. To transform the original scores into attention weights in the form of a probability distribution through exponential normalization, V is a value vector. After weighted summation, the physical semantics are aggregated into the final features for representation.

[0065] Step S403: Input the hidden layer state vector into the preset motion decoding network for mapping and decoding to generate the target pose sequence and compliant impedance parameters of the end effector.

[0066] The hidden layer state vectors represent a high-dimensional abstract semantic space and cannot directly drive the underlying physical entities. The system inputs the hidden layer state vectors into a pre-defined motion decoding network to perform dimensionality reduction mapping and decoding. The motion decoding network maps the abstract semantic vectors to Cartesian physical space, generating a 3D target pose sequence to guide the end effector's motion. The target pose sequence includes 3D absolute coordinates and spatial pose quaternions. Simultaneously, to cope with the complex rigid body contact mechanics feedback in industrial scenarios, the motion decoding network synchronously generates compliant impedance parameters that match the target pose sequence, achieving compliant adaptation to contact forces from the external physical environment. The compliant impedance parameters include the spatial stiffness matrix and damping matrix during environmental interaction.

[0067] Step S404: The target pose sequence and compliant impedance parameters are sent to the inverse dynamics solver. The inverse dynamics solver performs dynamic calculations based on a preset servo control cycle and converts them into a joint torque control sequence to drive the servo motor to perform physical actions.

[0068] The instruction frequency generated by the forward inference of the large model is typically much lower than the control frequency of the underlying electromechanical hardware. The system sends the target pose sequence and compliant impedance parameters to the underlying inverse dynamics solver of the embodied intelligent industrial robot via a hardware communication interface. The underlying inverse dynamics solver incorporates a physical dynamics constraint model based on the Lagrange equation. Using the received parameters as the ideal physical tracking target, and combining them with the prior physical properties of the embodied intelligent industrial robot body, such as link mass, moment of inertia, and Coriolis force, the underlying inverse dynamics solver performs inverse dynamics calculations within a preset servo control cycle at an extremely high frequency. After inverse dynamics calculations, the macroscopic kinematic commands are precisely converted into joint torque control sequences that conform to the underlying objective physical driving laws. Finally, the system transmits the joint torque control sequences to the underlying servo drivers of each joint to drive the servo motors to precisely execute physical actions, thus perfectly closing the pipeline from large model-side computational inference to objective physical control.

[0069] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the protection scope of the present invention. All data acquisition actions in this application are carried out in compliance with the relevant data protection laws and policies of the country where the application is located and with the authorization granted by the owner of the corresponding device.

Claims

1. A method for accelerating end-to-end inference of a large-scale embodied intelligent industrial robot, characterized in that, Includes the following steps: Step S1: Extract the target process identifier, collect visual feature images and end torque signals, and concatenate the mapping features of the target process identifier and end torque signals, global task markers, and visual feature markers of additional two-dimensional visual block coordinate sideband metadata to generate a multimodal input feature stream. Step S2: Extract the target process identifier of the current time step and the historical process identifier of the previous time step. When the target process identifier and the historical process identifier are inconsistent, determine that a process jump event has occurred. Locate the logical index corresponding to the global task marker in the input sequence of the previous process. Extract the key value tensor in the corresponding physical memory block in the paging key value tensor cache pool and divide the key value tensor into global process anchor point tensors. Solve the three-dimensional reachable envelope. Obtain the image feature tensor in the paging key value tensor cache pool that belongs to the previous process and corresponds to the visual feature marker. Read the two-dimensional visual block coordinate sideband metadata that is synchronously bound to the image feature tensor. Based on the two-dimensional visual block coordinate sideband metadata, back-project the image feature tensor into a three-dimensional scene point cloud. Divide the image feature tensor that is spatially detached from the three-dimensional reachable envelope into redundant transition tensors. Step S3: Release the physical memory block occupied by the redundant transition tensor. When updating the key-value cache logical block mapping table, delete the logical index entry corresponding to the redundant transition tensor from the key-value cache logical block mapping table to perform substantial sequence compression. For the surviving image feature tensor that has not been released, based on the continuous sequential index of the surviving image feature tensor in the compressed sequence, call the position encoding remapping operator to recalculate the rotation position encoding matrix of the surviving image feature tensor to perform continuous position index remapping. Asynchronously move the global process anchor tensor to the on-chip static random access memory through the direct memory access channel, and anchor the global process anchor tensor to the starting absolute position index of the generated multimodal input feature stream. Step S4: Generate a feature query vector sequence and a local key value tensor based on the multimodal input feature stream. Read the locked global process anchor point tensor and concatenate the local key value tensor to calculate the attention weight. Output the hidden layer state vector. Decode and generate the target pose sequence and compliant impedance parameters and send them to the inverse dynamics solver to solve for the joint torque control sequence to drive the motor.

2. The method for accelerating large-scale end-to-end inference of an embodied intelligent industrial robot as described in claim 1, characterized in that, Step S1 includes the following sub-steps: Step S101: Read the process state machine instructions issued in real time by the underlying programmable logic controller, decode the process state machine instructions, and extract the target process identifier that needs to be executed. Step S102: At the current time step, capture a visual feature image at a first sampling frequency and assign a first hardware timestamp, and at the same time capture an end torque signal at a second sampling frequency and record the corresponding second hardware timestamp sequence. The second sampling frequency is greater than the first sampling frequency. Step S103: Using the first hardware timestamp as the cutoff time, trace back to the sampling period corresponding to one of the first sampling frequencies to construct a causal sliding time window; Extract the end torque signal that falls within the causal sliding time window with the second hardware timestamp, perform feature aggregation on the extracted end torque signal, and calculate the torque peak value, torque mean value and torque temporal variance within the causal sliding time window respectively. Based on the feature aggregation results, generate the downsampled aligned torque feature vector.

3. The method for accelerating large-scale end-to-end inference of an embodied intelligent industrial robot as described in claim 2, characterized in that, Step S1 further includes the following sub-steps: Step S104: Map the target process identifier and the alignment torque feature vector through a preset linear projection layer respectively; The output dimension of the linear projection layer is constrained to be consistent with the feature dimension of the visual feature image output by the visual encoder, so as to obtain the process condition mark and force feature mark of the first dimension. The preset global task markers, process condition markers, force feature markers, and visual feature markers with additional two-dimensional visual block coordinate sideband metadata are extracted, concatenated along the sequence length dimension, and fused to generate a multimodal input feature stream suitable for attention mechanism computation.

4. The method for accelerating large-scale end-side inference of an embodied intelligent industrial robot as described in claim 3, characterized in that, Step S2 includes the following sub-steps: Step S201: In the forward inference process of the multimodal large language model, extract the target process identifier of the current time step and the historical process identifier of the previous time step, compare the target process identifier with the historical process identifier, and determine that a process jump event has occurred when the target process identifier and the historical process identifier are inconsistent. Step S202: In response to the process transition event, query the key-value cache logical block mapping table, locate the logical index corresponding to the global task marker in the input sequence of the previous process, extract the key-value tensor in the corresponding physical memory block in the paging key-value tensor cache pool based on the located logical index, and divide the key-value tensor into global process anchor point tensors.

5. The method for accelerating large-scale end-side inference of an embodied intelligent industrial robot as described in claim 4, characterized in that, Step S2 further includes the following sub-steps: Step S203: Query the preset industrial standard operating procedure knowledge graph, extract the motion limit parameters of each joint of the end effector that are mapped to the target process identifier, substitute each joint motion limit parameter into the forward kinematics model, and calculate the three-dimensional reachable envelope of the end effector under the target process through the boundary search algorithm.

6. The method for accelerating end-to-end inference of a large-scale embodied intelligent industrial robot as described in claim 5, characterized in that, Step S2 further includes the following sub-steps: Step S204: Obtain the image feature tensor belonging to the previous process in the pagination key value tensor cache pool, read the two-dimensional visual block coordinate sideband metadata that is synchronously bound to the image feature tensor, extract the pixel-level depth matrix of the corresponding image block from the depth information of the visual sensor based on the two-dimensional visual block coordinate sideband metadata, and perform coordinate system inverse transformation on the pixel-level depth matrix using the camera intrinsic and extrinsic parameter matrix to generate a three-dimensional scene point cloud located in the robot base coordinate system. Step S205: Perform pixel-level spatial difference operation on the point cloud of the three-dimensional scene in two consecutive frames to isolate the dynamic feature region where the spatial depth change is greater than a preset threshold. For discrete points within the dynamic feature region, the instantaneous three-dimensional velocity vector is calculated using the first-order forward difference method; By introducing a look-ahead window parameter, and based on the instantaneous three-dimensional velocity vector and the look-ahead window parameter, a linear manifold projection is performed on the discrete points within the dynamic feature region to obtain the projected trajectory manifold; At each time sampling point of the projected trajectory manifold, the three-dimensional spatial Euclidean distance between each time sampling point and the outer boundary of the three-dimensional reachable envelope is calculated using the signed distance field function, and the minimum value at all time sampling points is taken as the shortest safe distance. When it is determined that the 3D scene point cloud corresponding to the preset image feature tensor is completely out of the 3D reachable envelope at the current time step, and the shortest safe distance corresponding to the preset image feature tensor is greater than the preset safety tolerance threshold, the preset image feature tensor is divided into a redundant transition tensor.

7. The method for accelerating large-scale end-side inference of an embodied intelligent industrial robot as described in claim 6, characterized in that, Step S3 includes the following sub-steps: Step S301: In response to the classification results of the global process anchor tensor and the redundant transition tensor, the paging memory management interface is called through the edge computing unit to obtain the physical memory addressing base address and physical memory data block length of the redundant transition tensor mapped in the paging key-value tensor cache pool. Step S302: A release command is sent to the video memory controller through the paging memory management interface. The video memory controller releases the logical address binding of the redundant transition tensor in the key-value cache logical block mapping table according to the physical video memory addressing base address and the physical video memory data block length, and forcibly releases the physical video memory block occupied by the redundant transition tensor. When updating the key-value cache logical block mapping table, the logical index entries corresponding to redundant transition tensors are deleted from the key-value cache logical block mapping table. For surviving image feature tensors that have not been released, the rotation position encoding matrix of the surviving image feature tensors is recalculated by calling the position encoding remapping operator based on the continuous sequential index of the surviving image feature tensors in the compressed sequence.

8. The method for accelerating large-scale end-side inference of an embodied intelligent industrial robot as described in claim 7, characterized in that, Step S3 further includes the following sub-steps: Step S303: While issuing the release instruction, configure the source address of the direct memory access channel as the first storage address of the global process anchor tensor in the paging key-value tensor cache pool, and configure the destination address of the direct memory access channel as the second storage address in the on-chip static random access memory. Start the direct memory access channel to asynchronously move the global process anchor tensor from the first storage address to the second storage address. After the transfer is completed, when generating the multimodal input feature stream of the next target process, the global process anchor tensor is inserted as the first feature marker of the input sequence, and the absolute position index 0 is assigned to the global process anchor tensor. The subsequent real-time feature markers are then sequentially concatenated from the absolute position index 1.

9. The method for accelerating large-scale end-side inference of an embodied intelligent industrial robot as described in claim 8, characterized in that, Step S4 includes the following sub-steps: Step S401: In the subsequent reasoning stage of the target process, extract the multimodal input feature stream of real-time input, and generate the feature query vector sequence and local key-value tensor of the current time step through linear transformation; Step S402: Read the global process anchor tensor locked in the on-chip static random access memory through the internal data bus, concatenate the global process anchor tensor and the local key value tensor in the sequence length dimension, calculate the similarity matrix with the feature query vector sequence, perform attention weight allocation, and output the hidden layer state vector.

10. The method for accelerating large-scale end-side inference of an embodied intelligent industrial robot as described in claim 9, characterized in that, Step S4 includes the following sub-steps: Step S403: Input the hidden layer state vector into a preset motion decoding network for mapping and decoding to generate the target pose sequence and compliant impedance parameters of the end effector; Step S404: The target pose sequence and compliant impedance parameters are sent to the inverse dynamics solver. The inverse dynamics solver performs dynamic calculations based on a preset servo control cycle and converts them into a joint torque control sequence to drive the servo motor to perform physical actions.

Images