An edge-side inference method for a lightweight visual language action model

Abstract

This invention discloses a lightweight visual language action model for edge-side inference, specifically relating to the field of edge-side visual language inference technology in intelligent warehousing. It addresses the problem that existing edge-side visual language action models, under conditions of limited computing power and power consumption, struggle to adaptively allocate inference resources and stably constrain action boundaries according to scene changes, thus failing to balance operational safety and efficiency. By constructing constraint vectors, behavior vectors, scene complexity, and input quality scores on the edge according to a unified time synchronization, and using a scheduling model to control the number of feature fragments, the number of cross-modal alignment layers, action decoding accuracy level, and safety margin coefficient, combined with safety-related fragment selection, instruction fragment compression, and re-inference and safety action switching based on stability indicators, this method can still focus on safety-critical areas and constrain action boundaries even under limited computing power and power consumption. This solves the problem that existing edge-side visual language inference methods struggle to balance safety and efficiency under varying operating conditions.

Description

Technical Field

[0001] This invention relates to the field of edge visual language reasoning technology in intelligent warehousing, specifically to an edge reasoning method for a lightweight visual language action model. Background Technology

[0002] In scenarios such as warehousing and sorting, unmanned transport vehicles (ARTVs) generally employ forward-facing cameras, voice or text commands, and path planning and obstacle avoidance algorithms to achieve autonomous driving. Existing technologies employ two main approaches: one deploys visual recognition, voice understanding, and motion planning primarily in the cloud or on a host computer, uploading images and status data at a fixed frequency, with a central server processing and issuing action commands. This approach offers relatively ample computing power and algorithmic complexity, but is highly sensitive to network latency and link stability. If communication jitter or interruption occurs, the edge device can only rely on simple local safety policies, making it difficult to respond promptly to complex operating conditions. Another approach attempts to offload simplified visual recognition and control logic to the edge device, making local decisions through fixed resolution, fixed model structure, and pre-tuned speed limits and obstacle avoidance rules. This typically involves only triggering coarse-grained degradation strategies such as frame rate reduction or disabling certain functions using simple load or temperature thresholds. The inference link is relatively rigid, lacking detailed perception and adaptive capabilities regarding on-site conditions.

[0003] As visual language action models are increasingly used in robotics, some existing systems are beginning to introduce multimodal models on the edge to understand natural language commands such as "go to a certain storage location to pick up goods" and "avoid the forklift in front," and generate action sequences by combining them with camera footage. However, existing publicly available technologies generally have several common limitations: First, the allocation of inference resources largely relies on fixed configurations, at most simply switching between high and low-level models based on computing power usage or temperature, rarely integrating multi-source information such as chip computing power, power consumption limitations, task priority, historical action trajectories, scene complexity, and input quality to form unified constraints. In terms of time, there is also a lack of fine-grained slicing based on unified time synchronization, which makes it impossible for the model on the edge to adjust the feature retention range, cross-modal alignment depth, and action decoding accuracy as needed according to scene changes. Secondly, most safety-related decisions rely on local threshold triggers, such as triggering emergency stops based on the minimum obstacle distance in a single frame, or using simple "speeding is speed limit" rules for protection. They rarely integrate information such as the magnitude of action changes, the direction of actions over continuous periods, and the degree of deviation from verified safety trajectories into quantifiable stability indicators to drive a complete set of action replanning and safety action switching processes. Furthermore, they lack the ability to form a complete chain of evidence on the device side that is bound to the configuration version and model version, supporting post-event traceability and consistent reproduction across devices.

[0004] In summary, the core problem that exists in the deployment of visual language action models to the edge in warehouse sorting scenarios is the lack of a closed-loop mechanism based on a unified time base that can simultaneously perceive resource status, behavioral characteristics, scene complexity, and input quality, and adaptively adjust the inference granularity and action safety boundaries on a time scale, under conditions of limited computing power and power consumption and dynamic changes in operating conditions. This makes it difficult for the edge to achieve a stable balance between safety and operational efficiency in a timely manner, especially under complex operating conditions such as multiple vehicles traveling together and frequent changes in lighting and occlusion. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a lightweight on-device reasoning method for visual language action models, thereby resolving the problems mentioned in the background section.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a lightweight visual language action model edge-side inference method, comprising:

[0007] S1. Collect computing power usage, power consumption limits, task priorities, and action trajectories and environmental changes over several time periods on the edge device to generate constraint vectors and behavior vectors;

[0008] S2. Based on the image summarization, instruction representation and sensor quality index input evaluation model, the scene complexity score and input quality score are obtained;

[0009] S3. Input the constraint vector, behavior vector, scene complexity score and input quality score into the scheduling model, and output the number of feature segments, the number of cross-modal alignment layers, the action decoding accuracy level and the safety margin coefficient.

[0010] S4. Based on the number of feature segments, the feature segments are scored for security relevance during the visual encoding stage. Target region segments and risk region segments are retained first to form a set of feature segments that participate in the visual encoding operation.

[0011] S5. In the cross-modal alignment stage, compress action-irrelevant instruction segments according to the number of alignment layers. In the action decoding stage, decode safety-critical action components and non-critical action components according to the action decoding accuracy level and safety margin coefficient, and trim the amplitude.

[0012] S6. At the end of each time period, calculate the stability index based on the current action, historical actions and environmental changes. When the stability index exceeds the threshold determined by the safety margin coefficient, perform re-inference and switch to a safe action sequence.

[0013] Furthermore, S1 includes:

[0014] The edge devices operate in fixed-length time periods driven by a unified timing clock. At the beginning of each time period, the percentage of computing power used, the upper limit of power consumption, the temperature margin, and the current task priority are collected and combined to form a constraint vector.

[0015] Within an observation window covering a preset number of consecutive time periods, statistical characteristics of the action and environment are calculated based on the changes in the action trajectory and the changes in obstacle distance in each time period, and then combined to form a behavior vector.

[0016] The runtime management library uses an idempotent key consisting of device identifier, time period number, and task number as an index to associate and store constraint vectors and behavior vectors with the corresponding idempotent keys.

[0017] Furthermore, S2 includes:

[0018] The edge device captures images from the front-view camera under a unified time clock, and forms an image summary through resolution compression and channel compression.

[0019] Obtain instructions from at least one of the voice channel and text channel, and map them into instruction representations according to a preset vocabulary;

[0020] The sensor quality monitoring unit calculates sensor quality indicators based on image summarization;

[0021] The edge device aligns the image summary, instruction representation, and sensor quality indicators with the constraint vector and behavior vector according to the unified time synchronization and inputs them into the evaluation model to obtain the scene complexity score and input quality score. The score is then written into the runtime management library using the idempotent key as the index.

[0022] Furthermore, S3 includes:

[0023] The edge device reads the constraint vector, behavior vector, scene complexity score and input quality score associated with the idempotent key from the operation management library based on the idempotent key, and combines the four into a scheduling feature vector according to the preset field order and inputs it into the scheduling model.

[0024] The scheduling model outputs the number of feature fragments, the number of cross-modal alignment layers, the action decoding accuracy level, and the safety margin coefficient. The output results, along with the scheduling model version number and the configuration table version number, are written to the runtime management library and indexed by an idempotent key.

[0025] Furthermore, the scheduling model scores the scheduling feature vector and the preset gear combination based on the configuration table, and selects the gear combination with the highest score as the number of feature segments, the number of cross-modal alignment layers, the action decoding accuracy level, and the safety margin coefficient.

[0026] If no scheduling result is obtained within the preset delay limit, the end device uses the scheduling result recorded in the previous time period, increases the safety margin coefficient level, decreases the feature fragment quantity level and the action decoding accuracy level, and records the rollback reason and associated idempotent key in the evidence chain of the operation management library.

[0027] Furthermore, S4 includes:

[0028] The end-side device marks the feature segments that overlap with the target detection box as target area segments based on the target detection results, and marks the feature segments that overlap with the risk area as risk area segments based on the preset risk area. It also calculates the safety relevance score of each feature segment based on the overlap ratio and the spatial distance to the emergency stop event and the close-range passing event.

[0029] The end-side equipment sorts the feature segments according to the number of feature segments in the scheduling results and selects feature segments whose number is no greater than the maximum number of feature segments in that level as the set of feature segments to participate in the visual coding operation. When the number of target area segments is less than the maximum number of feature segments in that level, risk area segments are used to make up the difference.

[0030] Furthermore, S5 includes:

[0031] The end-side device parses the instruction representation into action target segments, safety constraint segments, and action-independent instruction segments based on the instruction phrase table;

[0032] Weights are assigned to action-independent instruction fragments based on behavior vectors and scene complexity scores, and the fragments are compressed into summaries. The summaries, along with action target fragments, safety constraint fragments, and a set of filtered feature fragments, are then input into a cross-modal alignment network whose number of cross-modal alignment layers is determined by scheduling results.

[0033] The end-side device decodes the joint representation to obtain action components based on the action decoding accuracy level and safety margin coefficient in the scheduling results. The amplitude of safety-critical action components is limited according to the safety margin coefficient, and the action components, together with the configuration table version number, are written into the operation management library and indexed by an idempotent key.

[0034] Furthermore, S6 includes:

[0035] At the end of each time period, the end-side device reads the current safety critical action components, historical safety critical action components, and environmental change records from the operation management library by pressing the idempotent key at the unified time synchronization time.

[0036] Within the observation window, the variation amplitude and directional consistency of safety-critical action components are statistically analyzed, and the action sequence is aligned with the safety trajectory template to obtain the trajectory deviation.

[0037] The end-side device combines the amplitude of motion change, directional consistency, and trajectory deviation into a stability index based on preset weights, and retrieves the stability threshold from the stability threshold configuration table based on the current safety margin coefficient, and compares the stability index with the stability threshold.

[0038] Furthermore, when the stability index exceeds the stability threshold, the edge device writes the stability exceedance mark into the constraint vector in the current time period, increases the safety margin coefficient level, and calls the scheduling model to obtain the updated feature fragment quantity level, cross-modal alignment layer number and action decoding accuracy level. Under the same observation information, it performs re-inference and replaces the original action component.

[0039] When the number of re-inference attempts reaches the upper limit within the observation window and the stability index still exceeds the stability threshold, the edge device selects a safety action sequence from the safety action library that meets the current speed, obstacle distance, and scene category, issues it to the chassis controller, and writes the stability index, stability threshold, number of re-inference attempts, safety action template identifier, and configuration version number into the operation management library and links them into the evidence chain.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] 1. By constructing constraint vectors, behavior vectors, scene complexity scores, and input quality scores on the edge according to unified timing, and controlling the number of feature fragments, the number of cross-modal alignment layers, the action decoding accuracy level, and the safety margin coefficient through a scheduling model, and combining feature fragment selection based on safety relevance scoring, cross-modal instruction fragment compression, and re-inference and safety action switching triggered by stability indicators, it can continuously focus on safety-critical areas, dynamically constrain action boundaries, and maintain stable action trajectories even under limited computing power and power consumption conditions. This solves the problem that existing edge-side visual language action reasoning cannot adaptively balance safety and operational efficiency with changes in operating conditions.

[0042] 2. By using idempotent keys as the main thread to connect constraint vectors, behavior vectors, scoring results, scheduling results, feature fragment screening records, action components and stability assessments, and various configuration version numbers in the operation management library, a replayable evidence chain is constructed and used in conjunction with the gear configuration table corresponding to the on-site safety specifications. This enables the entire end-side inference process to have a unified standard, traceable decision-making basis, and replicable deployment method under different equipment and operating conditions. This facilitates engineers to conduct online debugging, operation monitoring, and post-event review, and supports rapid adaptation to different warehousing production lines through configuration adjustments without changing the core model structure. Attached Figure Description

[0043] Figure 1 This is a flowchart illustrating the edge-side inference method for a lightweight visual language action model according to the present invention. Detailed Implementation

[0044] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0045] Example: Figure 1 A flowchart illustrating an edge-side inference method for a lightweight visual language action model according to the present invention is provided. The edge-side inference method for a lightweight visual language action model includes:

[0046] S1. Collect computing power usage, power consumption limits, task priorities, and action trajectories and environmental changes over several time periods on the edge device to generate constraint vectors and behavior vectors. The specific implementation is as follows:

[0047] In the warehouse sorting channel scenario, each unmanned transport vehicle equipped with end-side equipment operates in discrete time periods under the drive of a unified time clock. The end-side equipment preferably includes a chip monitoring module, a task queue management module, an operation management library, and a communication unit connected to the chassis controller and environmental perception module, which is used to generate the constraint vector and behavior vector of the current time period at the beginning of each time period.

[0048] The edge device is a local computing unit installed on the transport vehicle body, with defined computing power, power consumption limits, and storage capacity. It can complete the inference and related calculations of visual language action models without relying on a remote server. The time period is a fixed-length control cycle divided according to a unified timing clock. Each time period completes one round of perception, decision-making, and action issuance. The time period length can be set to the tens of milliseconds, preferably not less than twenty milliseconds and not more than one hundred milliseconds, to balance real-time performance and edge computing power. The unified timing clock is a time base signal aligned within the edge device or the vehicle network, used to time-align the measurement values ​​of the same time period between different modules.

[0049] At the start of each new time period, the chip monitoring module accumulates statistics on the busy / idle status of the chip's computing cores according to the start and end times of that time period. The percentage of computing power occupancy is preferably defined as the proportion of the time the computing core is in working state during that time period to the total time of that time period, expressed in the range of 0% to 100%. The upper limit of power consumption is determined by the rated power supply capacity given by the power supply module at the time of deployment, expressed in watts, and is adjusted as needed during operation based on temperature and aging conditions. The temperature margin is defined as the difference between the chip's maximum allowable operating temperature and the current actual temperature of the chip. The maximum allowable operating temperature is preferably determined by multiplying the maximum operating temperature specified in the chip's datasheet by a safety factor less than one. The safety factor can be set to a fixed proportion less than one, for example, it can be set to 90% of the nominal value, and locked in the configuration file by the version number.

[0050] The task queue management module maintains a list of pending handling tasks and configures a priority for each task. The priority can be set to several discrete levels, preferably from level one to level five, where level one represents the highest priority and is used for urgent orders or critical channel tasks, and level five represents the lowest priority and is used for daily replenishment tasks.

[0051] The chassis controller collects vehicle speed, steering angle and vehicle position in the warehouse coordinate system at a fixed rhythm in each time period to form motion trajectory information for that time period; the environmental perception module measures the distance to surrounding shelves, pedestrians and other vehicles, as well as the displacement of the current target goods relative to the vehicle, through forward-looking or surround-view sensors. The distance and displacement are preferably expressed in meters, and the change compared to the previous time period is calculated at the time period boundary.

[0052] Since the sampling times of different modules may vary slightly, the end-side device preferably uses a unified time synchronization time as a reference to time-align the data of each module before forming the constraint vector and behavior vector. That is, within a time period, the sampled value closest to the start or end time synchronization time of the time period is selected as the representative value for each statistical quantity, and the corresponding sampling time is recorded in the operation management library. When a module fails to provide a valid measurement in the current time period, it can be set to use the valid value of the nearest adjacent time period for supplementation. The nearest adjacent time period preferably refers to the time period before the current time period. If there is no valid value in the previous time period, it can be set to use the valid value of the next time period and mark the supplementation mark on the field to distinguish it from the directly measured value.

[0053] For measurement points that clearly do not conform to the physical constraints of the vehicle, the end-side device can set noise judgment rules. For example, when the linear velocity recorded in a certain period is greater than the preset maximum vehicle speed limit multiplied by a preset multiplier factor, it is judged as abnormal. The multiplier factor is a fixed value greater than one, preferably set to two. Alternatively, when the linear velocity in that period is greater than twice the average value of the velocity change centered on the current period and covering the previous three and the next three periods, it is judged as abnormal. When the obstacle distance is less than zero, it is also directly judged as abnormal.

[0054] The maximum speed limit is given in the vehicle manual during deployment and recorded as a configuration parameter. Measurements deemed abnormal are not directly involved in vector construction. Instead, a sliding window centered on the current time period and covering the previous three and next three time periods is preferred. The average value of measurements in other time periods within the window is used as a substitute. At the same time, the original measurement value and the flag of the substitution behavior are retained in the operation management library for post-event review.

[0055] After alignment, padding, and noise smoothing, the edge device organizes resource-related quantities into a constraint vector for the current time period according to a pre-agreed field order and unit. This constraint vector preferably includes at least the percentage of computing power utilization, power consumption limit, temperature margin, and current task priority. It can also be expanded to include remaining storage space on the edge device, the number of parallel tasks, etc., as needed. The behavior vector reflects the changing characteristics of actions and the environment over a short period. It is preferably constructed based on an observation window covering several consecutive time periods. The length of the observation window is configurable. To balance sensitivity to sudden local changes and the grasp of overall trends, the number of time periods included in the window is preferably no less than five and no more than thirty, typically set to ten consecutive time periods.

[0056] At the start of each new time period, the end-side device reads the motion trajectory records and environmental change records of all time periods within the observation window from the operation management library. It arranges the displacement, steering angle change, and obstacle distance change of each time period in chronological order within the window, calculates the overall average speed, speed extremes, cumulative displacement between the end and beginning of the window, and the minimum and magnitude of obstacle distance. These statistics are then arranged into a behavior vector in a fixed order. The average speed can be set as the sum of the linear velocities recorded for each time period within the window, divided by the number of time periods within the window. The speed extremes are the maximum values ​​among the linear velocities mentioned above. The cumulative displacement is the distance between the position coordinates recorded at the end of the window and the position coordinates recorded at the beginning of the window. The minimum obstacle distance is the minimum obstacle distance recorded for all time periods within the window. The magnitude of obstacle distance change is the absolute value of the difference between the obstacle distance at the end of the window and the obstacle distance at the beginning of the window. In some scenarios that are more sensitive to extreme values, the average values ​​of these statistics can be replaced with the median values ​​without changing other steps to achieve an equivalent effect.

[0057] Those skilled in the art can directly construct behavior vectors based on the recorded physical quantities without additional reasoning. To ensure accurate access to the constraint vectors and behavior vectors of the current time period in subsequent stages, the runtime management library adopts a recording method with idempotent keys as the primary index. The idempotent key is composed of a device identifier, a time period number, and a task number. The device identifier corresponds to the unique number of each transport vehicle, the time period number is formed by sequentially accumulating from the system startup based on a unified time clock, and the task number corresponds to the identifier of the transport task currently being executed in the task queue management module.

[0058] At the start of each time period, the edge device forms an idempotent key based on the current device identifier, time period number, and task number. It then checks the runtime management library to see if a corresponding record already exists. If no record exists, a new record entry is created and the constraint vector and behavior vector formed in this time period are written into it. At the same time, the configuration numbers of the current sampling period, observation window length, noise judgment threshold, and padding strategy are also written into it, so that the vector composition logic can be reconstructed under the same configuration environment in the future.

[0059] If multiple record requests from different sources occur within the same time period due to retry mechanisms or repeated communication triggers, the end device decides whether to overwrite the original record based on a preset integrity priority rule. This rule can be set as follows: First, compare the number of valid fields. If the number of valid fields in the new record is greater than that in the old record, it is considered more complete. If the number of valid fields is the same, then compare the number of fields marked with padding. If the number of padding fields in the new record is less than that in the old record, it is considered more complete. If both of the above are the same, the record that arrives first can be retained. Overwriting the original record is only allowed when the new record is determined to be more complete. At the same time, the overwriting time and source module identifier are added to the record entry, and this overwriting relationship is linked into the evidence chain.

[0060] The evidence chain is a logical chain in the operation management library that connects key records in order of time period number. It is used to track the entire process of each time period from the original measurement to the constraint vector, behavior vector and then to the subsequent scheduling and action decision. This allows engineers in the field to start from the idempotent key and trace back along the evidence chain to the computing power usage, temperature margin, task priority, trajectory statistics and environmental change statistics at that time when reviewing the on-site working conditions or secondary development. Under the same configuration version, the same vector composition can be obtained, thereby reproducing the decision premise at that time.

[0061] Preferably, the high load level and the safety level can be configured as a fixed set of levels during the deployment phase based on the hardware capabilities of the edge device and the maximum allowable load on site, and fixed in the configuration file with a version number. When the percentage of computing power usage is lower than the upper limit of the preset high load level and the temperature margin is higher than the lower limit of the preset safety level, it can be considered that the resources are relatively abundant. When the task priority is at level one or level two, it can be considered that the task urgency is relatively high.

[0062] On a typical straight sorting channel, the time period length can be set to about fifty milliseconds, the observation window covers ten consecutive time periods, the computing power usage percentage usually falls between 30% and 50% during warehouse off-peak hours, and can approach 70% during peak hours, the temperature margin can be maintained above 10 degrees Celsius when the ambient temperature is moderate, and the task priority can be marked as level 1 for urgent orders, level 3 for ordinary replenishment, and level 5 for low-impact tasks.

[0063] Under a representative working condition, the transport vehicle passes through the channel at a medium speed. Over ten consecutive time periods, the displacement changes approximately at a constant speed, the steering angle changes little, and the obstacle distance changes smoothly. The behavior vector calculated by the end-side device according to the above rules shows that the average speed is in the middle range, the extreme speed is close to the average value, the cumulative displacement is close to the displacement estimated by the average speed within the window over ten time periods, and the minimum obstacle distance is higher than the preset safety distance. The constraint vector reflects that the computing power occupancy is lower than the preset high load level, the temperature margin is in the middle to high range, and the current task priority is high. Under the same hardware platform and the same configuration version, when another engineer constructs the constraint vector and behavior vector according to the above units, rhythms, and rules, the same vector content can be obtained under the same working condition. Thus, the same preconditions can be obtained when calling the scheduling model and visual language action model in the future, realizing the reusability of the entire method.

[0064] S2. Based on the image summarization, instruction representation, and sensor quality index input evaluation model, the scene complexity score and input quality score are obtained. The specific implementation is as follows:

[0065] In the warehouse sorting channel scenario, in order to provide a scenario complexity score and input quality score that can be called by the subsequent scheduling model at each time period on the edge device, the edge device preferably completes image summary construction, instruction representation acquisition and sensor quality index summary under the constraint of a unified timing clock, and maps the above information into two scores within a fixed value range through the evaluation model.

[0066] The forward-facing camera captures raw images at a fixed rhythm within each time period. To reduce the computational burden on the edge device, it can be configured to select a frame near the start of the time period. This frame is then compressed using resolution and channels to generate an image summary. Resolution compression preferably involves scaling the raw image to a preset number of rows and columns. This number is configured during deployment based on edge computing power and field-of-view requirements and is fixed with a version number. Channel compression preferably retains only the luminance channel. In another implementation, the color channels can be linearly combined into a single-channel image with preset weights, while maintaining consistency in other steps. If necessary, it can be configured to retain only the region of interest covering the front aisle and pallet area to highlight areas associated with the handling action. The voice or text channel provides the current wheel command near a unified timing point. The edge device regularizes the command into a fixed-length command representation. This command representation preferably maps the command content into a multi-dimensional numerical vector according to a preset vocabulary or phrase list, ensuring consistent representation of the same type of command across different time periods.

[0067] The sensor quality monitoring unit quantifies the availability of camera and voice sensor signals in each time period, preferably providing sensor quality indicators such as brightness, contrast, blur level, noise intensity, and frame drop ratio. Brightness can be set as the average gray level of all pixels in the image summary; contrast can be set as the variance of the gray level of all pixels in the image summary relative to this average; blur level can be set as the reciprocal of the magnitude of the gray level difference between adjacent pixels in the image summary over the entire image; noise intensity can be set as the average of the mean square value of the gray level difference at the same pixel position in the image summary covering five time periods, centered on the current time period and covering the two preceding and following time periods; and frame drop ratio can be set as the ratio of the difference between the expected number of frames acquired and the actual number of frames acquired within a predetermined time window to the expected number of frames acquired.

[0068] The aforementioned image summaries, instruction representations, and sensor quality indicators are aligned with the constraint vectors and behavior vectors of the current time period at the time period boundary using the same alignment method according to the unified time synchronization time. That is, within the time range of a time period, the sampled value closest to the time synchronization time of the time period boundary is selected as the representative value of the current time period for each statistical quantity, and the time synchronization time corresponding to the sample is recorded in the operation management library so as to facilitate subsequent backtracking along the evidence chain and reconstruction of features under the same conditions.

[0069] The evaluation model is a lightweight scoring model deployed on the edge device. Its function is to combine image summarization, command representation and sensor quality indicators into a set of feature quantities at each time period, and give two scalars: scene complexity score and input quality score. The scene complexity score is used to reflect the number of target objects, the distribution of obstacles, the occlusion situation and the structural complexity of the area related to the current command in the current image. The input quality score is used to reflect the clarity and completeness of the current image and command and the working status of the sensor.

[0070] Before deployment, the evaluation model can determine its internal parameters through offline calibration data. During offline calibration, samples covering different warehouse layouts, lighting conditions, and workloads are collected. For each screen and corresponding instruction, the expected complexity and quality levels are given manually or by the upper-level system. These samples are then used to adjust the evaluation model parameters so that scenes with dense shelves, close obstacles, scattered target distribution, or significant movement receive higher scene complexity scores, while scenes with sparse shelves, open aisles, and single, relatively static targets receive lower scene complexity scores. Periods with moderate brightness, suitable contrast, low blur, low noise, and low frame drop rate receive higher input quality scores, while periods with excessively dark or bright brightness, low contrast, high blur, high noise, or high frame drop rate receive lower input quality scores.

[0071] To facilitate the unified use of scheduling models and monitoring logic, the internal value range of scenario complexity scores and input quality scores is fixed as decimals between zero and one in the configuration file during the deployment phase. The scoring scale and grading boundaries are recorded with the configuration version number and remain unchanged during runtime. When it is necessary to display the scores as a percentage, it can be set to obtain derived values ​​between zero and one hundred by converting them at a fixed ratio in the runtime management library, but the derived values ​​do not participate in the internal calculation.

[0072] Within each time period, the edge device combines the image summary, instruction representation, and sensor quality index consistent with the idempotent key of this time period into a feature vector according to a preset order and sends it to the evaluation model. The evaluation model gives a scene complexity score and an input quality score. The edge device associates the two scores obtained in this time period with the corresponding idempotent key and writes them into the operation management library. At the same time, it writes the evaluation model version number, the score scale configuration version number, and the configuration version number of the constraint vector and behavior vector of this time period into the record entry, so that the score results at that time can be reproduced under the same configuration environment when backtracking along the evidence chain.

[0073] To ensure the robustness of the scoring under sensor status fluctuations, when sensor quality indicators are incomplete for a certain period, such as when blur level or noise intensity cannot be obtained due to short-term occlusion or bandwidth limitations, the edge device can be set to use the same sensor quality indicators recorded in the previous period as a substitute. In the current period's record, the fields using the substitute values ​​are marked as supplementary records. At the same time, the scene complexity score is downgraded by one level according to the preset level table to leave a safety margin. The level table is configured according to business needs during the deployment phase and is fixed with a version number. The relevant degradation decision, along with the current configuration version number, is written into the operation management library and linked to the evidence chain for subsequent analysis of the applicable boundaries and operating conditions in the field. When sensor quality indicators are continuously missing for several consecutive periods and the substitution ratio exceeds the preset limit, the input quality score can be restricted to the low to medium level range, and a prompt message can be sent to the upper-level monitoring system through the communication channel to guide manual inspection of the camera or network status.

[0074] In actual production lines, the evaluation model is applicable to various warehouse layouts such as straight aisles, cross aisles, and multi-rack areas. The installation position of the forward-looking camera can be set to cover the aisle in front of the vehicle and the pallet area. Voice or text commands mainly include target storage location, cargo type, and motion constraint information. The sensor quality monitoring unit integrates with the camera driver, voice acquisition module, and network stack to statistically analyze various indicators according to a fixed rhythm.

[0075] In a representative scenario, the image summary resolution is set to 32x32, brightness and contrast are within a predetermined normal range, the average gradient magnitude corresponding to blur level is high, the mean square difference value corresponding to noise intensity is low, and the frame drop ratio is close to zero. In this scenario, the transport vehicle travels in a straight line between shelves, the shelves are neatly arranged, the aisle width is fixed, the target pallet position is stable, and the instruction representation includes a single location and simple handling requirements. Under this scenario, the evaluation model can give a lower-middle scene complexity score and a higher-middle input quality score in the aforementioned configuration version. In another representative scenario, the transport vehicle enters an area where forklifts and manual transport vehicles coexist, multiple dynamic targets exist simultaneously in the image, some shelf ends block the aisle, there is a significant difference in lighting on both sides, local overexposure areas appear in the camera pixels, and the noise intensity increases, the frame drop ratio rises, and the instruction representation includes multiple target order and priority information. In this case, the evaluation model can give a higher scene complexity score and a lower input quality score in the same configuration version. Based on this, those skilled in the art can, according to the actual output of the forward-looking camera and voice channel, and in conjunction with the specific implementation of the sensor quality monitoring unit, align the image summary, instruction representation, and sensor quality indicators by time period and send them into the evaluation model. Under the premise of a given scoring scale and parameter version, the process of constructing the scene complexity score and the input quality score can be reproduced, and consistent scoring results can be obtained under the same working conditions and configuration environment. This provides a stable and traceable basis for the quantification of complexity and quality for subsequent scheduling models and action decisions.

[0076] S3. Input the constraint vector, behavior vector, scene complexity score, and input quality score into the scheduling model, and output the number of feature segments, the number of cross-modal alignment layers, the action decoding accuracy level, and the safety margin coefficient. Specifically, this is implemented as follows:

[0077] In a warehouse sorting channel scenario, after updating the constraint vector, behavior vector, scene complexity score, and input quality score, the edge device performs a scheduling process once per time period to determine the resource allocation and safety boundaries of the visual language action model on the edge during that time period. Preferably, the edge device reads four types of quantities associated with the idempotent key corresponding to the current time period from the runtime management library: constraint vector, behavior vector, scene complexity score, and input quality score. The constraint vector reflects resource and task constraints such as the current chip computing power occupancy percentage, power consumption limit, temperature margin, and task priority. The behavior vector reflects the statistical characteristics of changes in action trajectories and obstacle distances over recent time periods. The scene complexity score reflects the complexity of the current image structure and target distribution. The input quality score reflects the clarity and completeness of the image and instructions.

[0078] The end-side device normalizes the above four types of quantities according to the pre-agreed field order and numerical range, and then combines them into a scheduling feature vector, which is sent to the scheduling model. The scheduling model is deployed on the end-side device and is used to provide four scheduling results within the time delay limit specified by the end-side device: the number of feature segments, the number of cross-modal alignment layers, the action decoding accuracy level, and the safety margin coefficient.

[0079] The number of feature fragments limits the number of visual feature fragments that can be retained and participate in fine-grained calculations in subsequent visual encoding stages. It is preferably set to several preset integer levels, such as dozens, hundreds, or even millions of fragments, to dynamically allocate visual details under different operating conditions when edge computing power is limited. The number of cross-modal alignment layers limits the depth of alignment operations between visual features and instruction representations. It can be set to several preset layer levels, such as few layers, medium layers, and multiple layers, corresponding to different computational overheads and the ability to characterize subtle semantics. The action decoding precision level limits the level of numerical precision used in subsequent action decoding stages. The algorithm is divided into several discrete levels, such as coarse, medium, and fine. The coarse level uses a larger step size for encoding only critical motion components in order to achieve lower latency. The medium level is a compromise between critical components and some non-critical components. The fine level uses a smaller step size for all motion components to achieve higher accuracy. The safety margin coefficient is a real number in the range of zero to one. It is used to shrink or relax the motion boundaries in motion decoding and stability judgment. When the coefficient is close to one, it means that a larger safety margin is used and the motion range is tightened. When the coefficient is close to 0.5 or even lower, it means that the motion boundaries are appropriately relaxed to improve efficiency while meeting the basic safety specifications.

[0080] The above four types of scheduling results are all limited to a preset set or range of gears. The correspondence between each gear and the specific end-side hardware capabilities and the safety specifications of the warehouse production line is defined by a configuration table during deployment. The configuration table provides the acceptable upper limit of end-side inference latency, upper limit of chip power consumption, and safety boundaries that match the maximum speed and minimum safe distance of the vehicle, respectively, for the number of feature fragments for each gear, the number of cross-modal alignment layers for each gear, the action decoding accuracy level for each gear, and several safety margin coefficient ranges. After deployment, the configuration table is fixed with the configuration version number and recorded in the runtime management library, and will not be changed arbitrarily during the runtime.

[0081] The scheduling model can determine parameters through offline calibration. During offline calibration, multiple sets of operational data covering light load, heavy load, simple scenarios, and complex scenarios can be collected. For each set of data, engineers or the host system can provide the desired resource allocation strategy. For example, when computing power consumption is low, scenario complexity is moderate, and input quality is high, it is desirable to use a medium-to-high number of feature segments, a medium number of cross-modal alignment layers, and a medium level of action decoding accuracy to balance safety and efficiency. When computing power is close to the upper limit or temperature margin is low, even if the scenario complexity is high, it is preferable to reduce the number of feature segments or the number of cross-modal alignment layers to ensure that edge inference can be completed within the latency limit. When the scenario complexity score is high and the input quality score is low, the safety margin coefficient is increased and the action decoding accuracy level is appropriately reduced to avoid making aggressive actions when the information is unreliable.

[0082] The scheduling model can be set to score the combination of scheduling feature vector and candidate gear positions in the configuration table. In each time period, the highest-scoring combination that meets the current constraint vector and safety specifications is selected as the scheduling result for this time period, thus achieving a trade-off between resource constraints, scenario complexity and input quality.

[0083] To meet the real-time requirements of the endpoint, the endpoint device preferably sets a delay limit for the scheduling model that is related to the length of the time period. For example, the inference time of the scheduling model is limited to a certain proportion of the time period length. When the endpoint device calls the scheduling model, it sets a timeout for the inference. If the scheduling model fails to return all four scheduling results within the specified delay, in order to avoid the interruption of no scheduling scheme, it can be set to roll back and use the complete scheduling result recorded in the previous time period. The triggering reason of this rollback event, the timeout time, and the idempotent key of the rollback result are recorded in the runtime management library and the evidence chain, so as to facilitate subsequent analysis of endpoint performance bottlenecks and model load.

[0084] When the scheduling result of the previous period is significantly mismatched with the current constraint vector, such as when the current temperature margin or power consumption limit has exceeded the pre-configured safety threshold, it can be further set to forcibly increase the safety margin coefficient to a conservative level on the basis of backoff, and reduce the number of feature segments or the action decoding accuracy level. This is recorded in the evidence chain in the form of backoff plus degradation, so as to ensure that even if the scheduling model does not respond in time, the system's safety boundary will not be breached.

[0085] After each time period is scheduled, the edge device writes the number of feature segments, the number of cross-modal alignment layers, the action decoding accuracy level, and the safety margin coefficient, along with the scheduling model version number, configuration table version number, and generation time, into the runtime management library. It then associates these values ​​with the constraint vector, behavior vector, scene complexity score, and input quality score of the current time period through idempotent keys, and concatenates them into the evidence chain in the order of time period numbers. This chain is used to constrain the selection of feature segments in the subsequent visual encoding stage, the pruning of the number of alignment layers in the cross-modal alignment stage, and the control of action accuracy and safety boundaries in the action decoding stage.

[0086] In typical straight-line sorting channel conditions, the scheduling model can be configured to use a medium-range number of feature fragments, a medium-to-high number of cross-modal alignment layers, a medium-to-high level of action decoding accuracy, and a medium-to-high level of safety margin when the percentage of computing power occupancy is at a medium level, the temperature margin is at a medium-to-high level, the task priority is high, and the scene complexity score and input quality score are at a medium-to-high level. When the transport vehicle enters an area with dense shelves and many mixed vehicles, the scene complexity score increases significantly, and the input quality score decreases slightly due to lighting or noise. In this case, the scheduling model can be configured to use a higher number of feature fragments and a higher number of cross-modal alignment layers to enhance the capture of details, while increasing the safety margin and maintaining the action decoding accuracy at a medium level. This allows for decision support with richer visual information while ensuring safety.

[0087] Based on this, those skilled in the art can implement the scheduling model on the edge device according to the above-mentioned input / output caliber and gear configuration method. Through offline calibration and configuration table adjustment, the constraint vector, behavior vector, scenario complexity score and input quality score can be mapped to the scheduling result that conforms to hardware capabilities and on-site safety specifications, and the same resource allocation strategy can be stably reproduced under the same configuration version and operating conditions.

[0088] S4. Based on the number of feature segments, the feature segments are scored for security relevance during the visual encoding stage, and target region segments and risk region segments are retained first to form a set of feature segments participating in the visual encoding operation. The specific implementation is as follows:

[0089] In the warehouse sorting channel scenario, after the end device completes the writing of the scheduling result for the current time period, it enters the visual encoding stage. In order to continuously monitor safety-related areas under the constraints of limited computing power and time delay, it can be set to filter and rearrange the set of feature segments obtained by the basic division of the current screen based on the number of feature segments in the scheduling result.

[0090] Basic partitioning can be understood as sending the current frame image captured by the forward-looking camera into the edge visual feature extraction network at a unified time synchronization. A feature map is obtained under a fixed resolution and a fixed number of channels. The feature map is then divided into several non-overlapping grid units according to a preset grid. Each grid unit corresponds to a feature segment. Each feature segment is associated with a position rectangle in the original image to represent the spatial range covered by the segment.

[0091] The target detection results are provided by the detection module deployed on the edge. In the current frame, it provides one or more rectangular boxes for the areas where targets such as pallets, shelf ends, pedestrians, and other vehicles are located. The edge device can be set to mark feature segments that intersect with the feature segment location rectangle within the area covered by the rectangular box as target area segments. The risk area division is determined based on the warehouse layout and safety specifications during the deployment phase. It can be set to pre-define a set of polygonal or regular shaped areas within the braking distance in front of vehicles, intersecting passages, personnel passage areas, and protruding areas at the ends of shelves. During operation, feature segments that spatially intersect with these risk areas are marked as risk area segments and fixed in the configuration file with a version number.

[0092] The edge device calculates a security relevance score for each feature segment within each time period. This score consists of three types of information: overlap degree, relative position, and historical security events. The overlap degree can be set as the ratio of the overlapping area of ​​the target detection box or risk region with the feature segment location rectangle to the area of ​​the feature segment location rectangle. This ratio is between zero and one. When the ratio is greater than a preset upper threshold, it is considered high overlap; when the ratio is between the upper and lower thresholds, it is considered medium overlap; and when the ratio is lower than the lower threshold, it is considered low overlap. Different overlap levels correspond to different base scores. The upper and lower thresholds are locked as configuration parameters with version numbers during the deployment phase.

[0093] Relative position can be used to distinguish between segments located near the center of the vehicle's travel path and segments far from the travel path at the same overlap level. The vehicle's travel path can be represented as a centerline in the vehicle coordinate system. The end-side device can be set to assign higher weights to segments whose distance from the centerline in the lateral direction of the position rectangle center point is less than a preset distance threshold. This distance threshold can be set to an equivalent distance of several grid cell widths and recorded in the configuration file. Segments whose distance from the centerline is greater than this distance threshold have reduced weights.

[0094] The end-side device can also be configured to record the location index and occurrence time of emergency stop events and close-range passing events in the operation management library in the form of association with idempotent keys. An emergency stop event can be defined as the vehicle speed dropping from above a preset speed threshold to close to zero within no more than two consecutive time periods and the control unit recording a braking action. A close-range passing event can be defined as the minimum distance to an obstacle recorded by the environmental perception module within a certain time period being lower than a preset safe distance while the vehicle still maintains a non-zero speed. The speed threshold and safe distance threshold are configured during the deployment phase and fixed with a version number.

[0095] When calculating the safety relevance score for the current time period, the end-side device searches for records of emergency stops or close passes within an observation window covering a preset number of consecutive time periods. The distance between the record and the center point of the rectangle where the feature segment is located is less than a preset neighborhood threshold. The length of the observation window is preferably ten consecutive time periods. The neighborhood threshold can be set as the equivalent distance of the diagonal length of several grid cells and recorded in the configuration file. When there are records of emergency stops or close passes that meet the distance condition within the observation window, an additional weight is added to the score of that segment, so that areas where dangerous approaches have occurred will maintain a high level of concern for a period of time in the future.

[0096] After calculating the security relevance score for each feature segment according to the above rules, the edge device sorts all feature segments from high to low scores. Then, combined with the feature segment quantity level specified in the scheduling result, it selects the batch of feature segments with the highest scores from the top of the sorting result. The number of selected segments does not exceed the maximum number of feature segments corresponding to the level in the configuration table. This batch is used as the feature segment set for the visual encoding operation in this time period. The unselected feature segments are no longer encoded with high precision. Instead, their features are summarized at the regional level. This can be set to average the features of multiple unselected feature segments in the same background area to form a small number of background summary vectors, which are used to provide the overall scene background for subsequent models without consuming too much computing power.

[0097] At the end of each time period, the edge device writes the feature segment identifiers, corresponding security relevance scores, actual number of selected segments, and the version number of the scoring rule currently used for the current time period into the operation management library, and attaches them to the evidence chain of the current time period through idempotent keys, so that when reviewing a certain action decision later, it can track which spatial areas were prioritized for encoding and what scoring rule was used at that time.

[0098] To ensure coverage of risk areas even when the number of target area segments is insufficient or their distribution is abnormal, a fallback rule can be set. When the number of segments marked as target areas in the current frame is less than the feature segment quantity level specified in the scheduling result, the end device first adds all target area segments to the candidate set, and then adds all risk area segments to the set according to their security relevance scores from high to low. When the total number of target area segments and risk area segments exceeds the upper limit allowed by the feature segment quantity level, a visual detail level with fewer channels and shallower intermediate layers can be selected from the configuration table. This allows for a slight reduction in the encoding precision of each segment without changing the spatial distribution of the area of ​​interest, ensuring that the overall encoding work is still completed within resource constraints. When the total number of target area segments and risk area segments is still insufficient to fill the feature segment quantity level, several segments can be added first from background segments whose lateral distance from the centerline of the travel path is less than the aforementioned distance threshold and which are not marked as target or risk areas. This ensures coverage of security-related areas while maintaining basic perception of the surrounding environment of the path.

[0099] In a set of representative operating conditions, such as a low-load scenario in a straight lane, hundreds of feature segments can be obtained after basic segmentation. Target detection only provides the single pallet in front and the ends of the shelves on both sides. The risk area is mainly concentrated within the braking distance directly in front of the vehicle. At this time, the segments with the highest safety relevance scores are concentrated in the pallet and the lane area in front. The feature segment quantity level in the scheduling result can be set to medium, and dozens of segments are selected for high-precision encoding during the visual encoding stage. In the scenario of intersecting lanes and mixed traffic, the number of target detection boxes and risk areas increases significantly, and multiple intersections and personnel areas appear in the picture at the same time. The feature segment quantity level in the scheduling result can be set to high, and the visual detail level is adjusted accordingly. Hundreds of segments are selected for encoding during the visual encoding stage. At the same time, the aforementioned rules ensure that all target area segments and most risk area segments are included in the selected set.

[0100] Based on this, those skilled in the art can implement the feature segment selection and resource allocation logic in the visual encoding stage on the edge device according to the above-mentioned feature segment division method, security correlation scoring rules and the segment selection strategy linked with the scheduling results, and stably reproduce the same encoding behavior under the same hardware configuration, parameter configuration and rule version.

[0101] S5. In the cross-modal alignment stage, compress action-irrelevant instruction segments according to the alignment layer number. In the action decoding stage, decode safety-critical action components and non-critical action components according to the action decoding accuracy level and safety margin coefficient, and trim the amplitude. The specific implementation is as follows:

[0102] In the warehouse sorting channel scenario, after the end-side device completes the feature segment selection in the visual encoding stage, it enters the cross-modal alignment stage. In order to retain as much semantic information closely related to action decision as possible within the limited number of cross-modal operation layers, it can be set to select the first few layers from the pre-deployed complete alignment layer sequence from bottom to top to participate in the alignment of this time period according to the number of cross-modal alignment layers in the current time period scheduling result. The complete alignment layer sequence is a set of fixed levels determined in the offline training stage of the model. The number of cross-modal alignment layers is recorded in the configuration table in the form of levels in the deployment stage and the specific level is selected by the scheduling model.

[0103] At the unified timing moment, the end-side device inputs the instruction representation obtained in the previous stage into the instruction parsing unit, and splits the instruction content into instruction fragments describing the action target, instruction fragments describing safety constraints, and action-irrelevant instruction fragments related to polite expressions and background descriptions according to semantic function. The action target fragment preferably includes fields such as target cargo location, cargo type, handling direction, and expected docking location. The safety constraint fragment preferably includes fields such as prohibited areas, speed limit requirements, and personnel priority. The action-irrelevant instruction fragments include polite expressions such as "please," "thank you," and "you've worked hard today," as well as background descriptions with low relevance to the current motion planning, such as "it's a bit dark" and "the warehouse is a bit messy."

[0104] To ensure consistency of the splitting rules across different batches, the edge device can set up an instruction phrase table during the deployment phase. This table lists common target statements, security statements, and background statements at the phrase level. The original instruction string is then mapped into several instruction fragments through keyword matching, position rules, and necessary part-of-speech tagging. Each fragment is marked with a category tag indicating whether it is "target-, security-, or action-independent" and its sequential index in the original instruction. The phrase table and its version number are stored in the configuration area of ​​the runtime management library.

[0105] After receiving the feature fragment set from the visual encoding stage and the aforementioned instruction fragment set, the edge device performs compression processing on the action-independent instruction fragments based on the behavior vector and scene complexity score of the current time period. The behavior vector reflects the fluctuations in speed, turning, and obstacle distance in recent time periods, and the scene complexity score reflects the complexity of the distribution of targets and obstacles in the current scene. The edge device can be configured to allow the retention of a small number of action-independent fragments related to the environmental background, such as fragments that indirectly affect the rhythm of the action, such as "Watch out for the forklift ahead" or "There is another batch of goods behind," when the complexity is high or the behavior changes drastically. When the complexity is high or the behavior changes drastically, all action-independent fragments are compressed into a short summary first. For example, multiple polite expressions are merged into a general label, and non-critical background descriptions are folded into an abstract label of "There is additional background information," thereby reducing the redundant semantic burden in cross-modal alignment.

[0106] Compression strategies can be implemented by assigning weights to action-independent segments. The weights are determined by the segment category, its position in the original instruction, and its relationship with the magnitude of velocity change and obstacle distance change in the behavior vector. The end device selects action-independent segments to be retained according to a preset weight threshold. The content of the remaining segments no longer participates in multi-level alignment operations and is only recorded in the language-side intermediate representation in the form of summary tags for this time period.

[0107] Subsequently, the edge device inputs the retained action target fragment, safety constraint fragment, and compressed action-independent instruction summary, along with the feature vector of the segment selected by the visual side in this time period, into the first few layers of the cross-modal alignment network. In each layer, semantic and spatial alignment operations are performed to gradually form a joint representation that carries target information, safety constraints, and key environmental features. When the number of alignment layers is small, only coarse-grained target and region matching is completed. When the number of alignment layers is large, the correspondence between semantic details and spatial details can be further characterized. The selection of these number of layers is constrained by the cross-modal alignment layer level in the scheduling results.

[0108] During the motion decoding phase, the edge device inputs the joint representation generated in the cross-modal alignment phase into the motion decoding unit. Based on the motion decoding accuracy level and safety margin coefficient in the scheduling results, the joint representation is mapped to the motion component set for the current time period and then pruned. The motion component set includes at least the vehicle's longitudinal speed component, lateral steering angle component, and, in scenarios with a robotic arm, the robotic arm's end-effector pose component. It can also be expanded to include additional components such as light cues, horn cues, and screen prompts, depending on the actuator configuration of the actual production line.

[0109] Preferably, during the deployment phase, the end-side equipment divides various motion components into safety-critical motion components and non-critical motion components according to their degree of impact on safety. Safety-critical motion components include longitudinal speed, lateral turning angle, and position parameters of the robotic arm end in personnel areas or narrow passages. Non-critical motion components include prompting actions that change within the specification range without affecting the safety baseline, as well as minor adjustments with small changes within the safety range.

[0110] The motion decoding accuracy level is given in the configuration table as a level. Different levels correspond to different numerical accuracy and update step size. For example, the speed component of the lower level uses a larger increment / decrement step size and a coarser quantization interval, while the higher level uses a smaller step size and a finer quantization interval. The safety margin coefficient takes a value in the range of zero to one. During the deployment phase, a mapping relationship is established with the maximum permissible speed, maximum permissible lateral acceleration, minimum safe distance, and robotic arm operating range in the on-site safety specifications. The closer the safety margin coefficient is to one, the smaller the corresponding upper limit of speed, upper limit of steering angle change rate, and upper limit of robotic arm pose offset, indicating a more conservative safety boundary. The closer the safety margin coefficient is to the preset median value, the closer the corresponding upper limit is to the maximum value allowed by the on-site specifications.

[0111] After obtaining the untrimmed motion components, the motion decoding unit can be configured to quantize each component according to the motion decoding accuracy level, mapping continuous values ​​to discrete grids of the corresponding level. Then, based on the safety margin coefficient and on-site safety specifications, amplitude trimming is performed on safety-critical motion components. When the longitudinal velocity component exceeds the speed limit corresponding to the current safety margin, it is truncated to the upper limit. When the change in steering angle exceeds the lateral change limit corresponding to the current safety margin, it is truncated to the upper limit. When the pose of the robotic arm end effector points beyond the safe workspace corresponding to the current safety margin, it is projected back to the vicinity of the safe workspace boundary. For non-critical motion components, trimming can be configured to only occur when they exceed the physical limits of the equipment or significantly deviate from the reasonable range of the scene. In other cases, the values ​​generated by the joint representation are retained to avoid unnecessary impact on efficiency.

[0112] The clipped motion components are attached to the current time period according to a unified time synchronization to form the final motion sequence of this time period. The end device writes the motion component vector of each time period, the motion decoding accuracy level used, the safety margin coefficient, and the corresponding configuration table version number into the operation management library, and links them into the evidence chain through an idempotent key. This allows the motion decision context at that time to be reconstructed under a unified standard when calculating stability indicators or evaluating system behavior during the on-site acceptance phase.

[0113] Under a set of representative working conditions, such as a straight sorting lane with only a single pallet ahead, a scene complexity score in the lower-middle range, moderate computational cost for constraint vector display, high temperature margin, and moderate task priority, the scheduling model can select a medium-range number of cross-modal alignment layers, a medium-range action decoding accuracy level, and a medium-range safety margin coefficient. In the cross-modal alignment stage, complete action target segments and safety constraint segments are preserved, and polite expressions are compressed into a single summary tag. In the action decoding stage, a medium speed and smooth turning are provided without triggering pruning. In cross-lane traffic with many mixed vehicles and a scene complexity score... In situations where the number of emergency stops increases significantly and the behavior vector reflects an increase in the number of emergency stops in the recent period, the scheduling model can select a higher number of cross-modal alignment layers, a lower action decoding accuracy level, and a higher safety margin coefficient. In the cross-modal alignment stage, more safety-related semantic segments are retained and action-irrelevant segments are completely compressed. In the action decoding stage, the longitudinal speed is limited to a lower upper limit and the instantaneous turning angle change is significantly restricted during the pruning process. At the same time, the pose of the robotic arm end effector is constrained to a safe space away from personnel and intersections. This achieves a traceable and reproducible balance between action decision granularity, inference resources, and safety boundaries under different working conditions.

[0114] Based on this, those skilled in the art can implement the specific processing flow of the cross-modal alignment stage and the action decoding stage under existing end-side hardware conditions, and stably obtain consistent action sequence output under the same configuration version and field conditions.

[0115] S6. At the end of each time period, calculate the stability index based on the current action, historical actions, and environmental changes. When the stability index exceeds the threshold determined by the safety margin coefficient, perform re-inference and switch to a safe action sequence. Specifically, the implementation is as follows:

[0116] In warehouse sorting scenarios, to form a traceable and quantifiable closed loop of motion stability at the edge, the edge device triggers a stability assessment process at a unified time at the end of each time period. Based on the motion components of the current time period, the motion components of several historical time periods within the same observation window, and the environmental change records corresponding to each time period, the device retrieves motion records and scheduling records matching the current device identifier and time period number from the operation management library in idempotent order. These are then arranged into a motion sequence within the same observation window, ordered from earliest to latest by time period number. The observation window can be set to cover several recent time periods, preferably ten consecutive time periods. Its length and sliding method are fixed in the operation management library during the deployment phase using configuration parameters and version numbers.

[0117] Within the observation window, the end-side device statistically analyzes the changes in safety-critical motion components such as longitudinal speed, steering angle, and end-effector pose. The magnitude of motion change can be set as the combination of the mean and maximum values ​​of the absolute values ​​of the differences between two adjacent time periods within a window. The direction of motion change in continuous time periods can be measured by the consistency of the signs of speed and steering changes within the window. For example, frequent alternation of positive and negative signs within the window is considered as directional instability. Environmental change records can be obtained from the aforementioned behavior vector and environmental perception module, and the degree of environmental disturbance is characterized by indicators such as minimum obstacle distance, distance change magnitude, and whether there are sudden stop events or close-range passing events.

[0118] To measure the deviation between the current action sequence and the verified safe trajectory, the edge device can establish a set of safe trajectory templates corresponding to typical warehouse routes and operating conditions during the deployment phase. Each trajectory template includes the vehicle's temporal position sequence, speed curve, and steering curve under standard operating conditions, and is registered in the operation management library along with the trajectory category, applicable scenario, and version number. During runtime, based on the current channel type, scenario complexity score, and behavior vector, one or more applicable templates are selected from the safe trajectory templates and aligned with the actual action sequence in the observation window in a unified coordinate system. The spatial deviation and speed deviation between the actual trajectory and the template trajectory within the window are calculated. The spatial deviation can be set as the mean and maximum of the minimum distance from each time point to the corresponding point of the template trajectory within the window, and the speed deviation can be set as the mean of the absolute value of the difference between the actual speed and the template speed within the window.

[0119] The stability index can be set as a single value synthesized from the magnitude of action changes, directional consistency, and trajectory deviation according to preset weights. The weights are determined through offline experiments during the deployment phase and fixed in the configuration file with version numbers. The greater the magnitude of action changes, the worse the directional consistency, and the more obvious the trajectory deviation, the higher the stability index. After calculating the stability index corresponding to the current time period, the end-side device, based on the safety margin coefficient in the scheduling result record of this time period, looks up a stability threshold corresponding to the current safety margin level, channel type, and scenario complexity range from the stability threshold configuration table. The stability threshold configuration table is set during the deployment phase according to on-site safety specifications and trial operation data, and is recorded in the operation management library with a configuration version number.

[0120] If the stability index for the current time period does not exceed the threshold, the action trajectory is considered to be within an acceptable range. The end device only appends the current stability index value, the referenced safety trajectory template identifier, and the threshold version number to the operation management library and links them to the evidence chain. If the stability index exceeds the threshold, the current action strategy is considered to be too aggressive under the current operating conditions or significantly affected by environmental disturbances. The end device calls the scheduling model again within the current time period, adds "stability exceeding the limit" as an additional constraint to the constraint vector, increases the safety margin coefficient by at least one level within the available level, and obtains the updated feature fragment quantity level, cross-modal alignment layer number, and action decoding accuracy level accordingly. Under the same observation information, a re-inference is performed to obtain a more conservative action plan. The new scheduling result and action components replace the original plan and the stability index is recalculated.

[0121] The number of times the re-inference process is triggered can be set to an upper limit within an observation window, preferably one to three times. The upper limit is fixed along with the configuration table and the version number. When the stability index still cannot fall below the threshold within the preset number of re-inferences, the end device will no longer try to improve stability by adjusting the model gear. Instead, it will select a safety action sequence from the local safety action library that matches the current speed, obstacle distance, and scene category and issue it to the chassis controller. During the deployment phase, the safety action library defines several types of safety action templates according to the on-site safety specifications, such as gradually decelerating to a safe low speed, stopping slowly in the current lane, standing still and activating audible and visual alarms, etc., and writes them into the operation management library with template identifiers and applicable conditions. Under the premise of meeting the braking distance and anti-rear-collision constraints, the end device selects a safety action sequence that meets the current constraints, sends the sequence to the chassis controller according to a unified timing rhythm, and transmits action instructions consistent with the safety status to the actuators such as robotic arms through the in-vehicle bus. At the same time, it reports the safety handling event corresponding to the current idempotent key to the upper-level scheduling system through the wireless network configured on the warehouse site, along with the stability index, the number of re-inferences, the safety action template identifier used, and the configuration version number. The entire stability assessment and safety handling process is recorded in the operation management library in the form of a chain of evidence, including the triggering cause, the gear used, the re-inference and the execution of safety actions, as well as all relevant configuration version numbers, for post-event review and on-site acceptance.

[0122] In a representative set of operating conditions, for example, the time period can be set to 20 milliseconds, the observation window covers the most recent ten time periods, the vehicle travels in a straight sorting channel at no more than half of the preset maximum operating speed, the scenario complexity score is about 0.6, the input quality score is about 0.8, the environmental disturbance is small, and the stability index remains below the threshold under the corresponding safety margin coefficient configuration, so there is no need to trigger re-inference. In another representative cross-channel operating condition, the sudden merging of other vehicles in front leads to an increase in the number of emergency stops, the magnitude of action changes and trajectory deviations increase significantly, and the stability index exceeds the threshold. The end-side device first performs one or two re-inferences by increasing the safety margin coefficient and reducing the action decoding accuracy level. The longitudinal speed and steering change magnitude are significantly tightened, and the index falls back below the threshold, so there is no need to enter the safety action library. If convergence is not possible under more extreme operating conditions, the slow stop or standby strategy in the safety action library will be triggered, thereby achieving a reproducible and auditable safety closed loop on the end side through the combination of stability index and safety action template without exceeding the preset maximum speed and minimum safety distance.

[0123] Those skilled in the art can, based on the above definitions and processes, implement stability index calculation, threshold lookup, re-inference triggering, and security action library invocation under existing end-side hardware and communication conditions, and obtain consistent security handling behavior under the same configuration version and operating conditions.

[0124] In the operating scenario shown in this embodiment:

[0125] In a warehouse scenario consisting of a straight sorting channel and intersecting channels, an unmanned transport vehicle equipped with end-side devices travels along a preset route from the inbound temporary storage area to the outbound platform under the constraint of a unified time synchronization clock. Throughout the transport process, a traceable operational link is formed on the end-side according to the aforementioned steps S1 to S6: When a task is issued, the task queue management module marks this transport task as a secondary priority. The corresponding task number, along with the device identifier and the initial time period number, forms an idempotent key. During several time periods at the beginning of the straight channel, the chip monitoring module detects that the computing power occupancy percentage is around 40%, the power consumption limit is operating at the rated value, and the temperature margin remains above 10 degrees Celsius. The chassis controller records that the vehicle is traveling in a straight line at a moderate speed with minimal changes in the steering angle. The environmental perception module records that the minimum obstacle distance is much higher than the preset safety distance and changes gradually. Based on this, the edge device constructs constraint vectors and behavior vectors for each time period and completes time alignment through unified timing. The noise judgment rule is not triggered. The average velocity and cumulative displacement calculated within the observation window are consistent with the linear uniform velocity behavior. The minimum obstacle distance and the magnitude of change indicate that the environmental disturbance is small. At the same time, the forward-looking camera starts to acquire images at the beginning of each time period and generates an image summary with a resolution of 32x32. The brightness, contrast, blur degree, noise intensity and frame drop ratio given by the sensor quality monitoring unit are all within the normal range. The voice or text channel contains only a target cargo location and simple handling instructions. Under this condition, the evaluation model gives a scene complexity score of about 0.4 and an input quality score of about 0.8 for each time period. The edge device writes the score, along with the idempotency key, the evaluation model version number and the score scale configuration version number, into the runtime management library and links it to the evidence chain.

[0126] After reading the constraint vector, behavior vector, and the two scores associated with the current idempotent key, the scheduling model combines them into a scheduling feature vector. Under the constraint that the upper limit of the edge latency is about half the length of the time period, it completes one inference and selects the medium-level feature fragment quantity level, the number of cross-modal alignment layers from few layers to medium layers, the medium-level action decoding accuracy level, and the medium-level safety margin coefficient. The runtime management library records the scheduling results and configuration version number for this time period.

[0127] In the visual encoding stage, the current frame is divided into hundreds of feature segments. The target detection module identifies the single pallet in front and the ends of the shelves on both sides. The risk areas are mainly concentrated within the braking distance directly in front of the vehicle. The end-side device calculates safety relevance scores based on overlap, relative position, and historical safety event records. After sorting, the top few dozen segments with the highest scores are selected as the feature segment set, and the remaining segments are aggregated into a small number of background vectors on both sides of the path according to region. In the cross-modal alignment stage, the instruction segments corresponding to the target location, cargo type, and speed limit requirements are completely preserved. Polite expressions such as "please" and "thank you" are folded into single summary tags and aligned with the visual side within the cross-modal alignment layer specified in the scheduling result. The fragments are aligned layer by layer to form a joint representation. During the action decoding stage, under the constraints of medium accuracy and medium safety margin coefficient, the joint representation is mapped to longitudinal speed and gentle steering angle. After quantization and pruning, the values ​​do not reach the upper limit of speed and the upper limit of lateral change. The end device sends the pruned action components to the chassis controller and writes them into the operation management library. During the stability assessment stage, the action change amplitude, directional consistency and trajectory deviation are statistically analyzed within the observation window covering the most recent ten time periods. A low stability index is calculated, which is lower than the stability threshold corresponding to the current safety margin coefficient. Therefore, re-inference and the safety action library are not triggered. Only the stability index value and the referenced safety trajectory template identifier are added to the evidence chain.

[0128] As the transport vehicle approaches the intersection, the environmental perception module detects a rapid decrease in the minimum distance to obstacles ahead and an increase in the magnitude of distance changes over a certain period of time. The speed and steering changes in the behavior vector also begin to increase. Multiple dynamic targets, such as forklifts and manual transport vehicles, appear in the pixels of the forward-looking camera. Overexposure patches and uneven lighting exist in some areas. The noise intensity and frame drop ratio given by the sensor quality monitoring unit increase slightly. In the new time period, the evaluation model increases the scene complexity score to about 0.75 and decreases the input quality score to about 0.6. The edge device records this score change along with the configuration version number in the operation management library.

[0129] Under the new scheduling feature vector, to meet computational constraints and safety regulations, the scheduling model automatically switches the feature fragment quantity level to a higher level, increases the cross-modal alignment layer level by one level, and increases the safety margin coefficient to a higher level, while slightly lowering the action decoding accuracy level. This prioritizes fine-grained encoding of safety-related areas and conservative action boundaries. In the visual encoding stage, under the new scheduling result constraints, hundreds of feature fragments are selected for encoding to ensure coverage of all target area fragments and most intersection risk area fragments. When necessary, the visual detail level is reduced to compress the resource consumption of a single fragment. In the cross-modal alignment stage, more safety constraint fragments related to personnel priority and prohibited areas are retained at a higher alignment layer level. All action-irrelevant instruction fragments are compressed into abstract labels with "additional background descriptions exist." In the action decoding stage, under the new safety margin coefficient constraints, the longitudinal speed and steering angle are subject to stricter amplitude clipping, so that vehicles automatically decelerate and limit instantaneous steering changes when approaching intersection areas. The position of the robotic arm end effector is restricted to a space range far away from the personnel passage area.

[0130] During a critical time period, a forklift suddenly merged into the lane, causing a surge in speed and obstacle distance changes in the behavior vector. The environmental perception module recorded that the minimum obstacle distance was approaching the preset safety distance. The stability index calculated during the stability assessment phase exceeded the stability threshold for the first time due to large changes in motion, poor directional consistency, and a significant increase in spatial deviation from the corresponding safety trajectory template. Based on this, the edge device wrote a "stability exceeded" flag into the constraint vector and called the scheduling model again within the same time period. This time, the safety margin coefficient was increased by one level, and the motion decoding accuracy level was reduced to a more conservative level. Without changing the observation information, a more conservative scheduling result and motion components were obtained. After recalculation, the stability index fell back below the threshold, therefore no intervention was required. A safety action library is generated, and the evidence chain records the triggering reasons for this re-inference, the scheduling gear adjustment, and the changes in stability indicators before and after the re-inference. When the entire transport task is completed, the evidence chain has connected the constraint vectors, behavior vectors, scene complexity scores, input quality scores, scheduling results, feature fragment selection records, action components, and stability indicators for each time period since the task started. Under the same hardware platform and configuration version, another person skilled in the art can start from any idempotent key and backtrack along the evidence chain to reconstruct the vector composition, scheduling decisions, visual encoding and cross-modal alignment behavior, as well as the stability assessment and re-inference process of the above key time periods under the same working conditions, thereby reproducing the behavior trajectory and safety control effect of the entire end-side visual language action model in the actual operating scenario.

[0131] All calculations involved in the embodiments are dimensionless numerical calculations, and the preset parameters and thresholds in the calculations are set by those skilled in the art according to the actual situation.

[0132] It should be noted that this invention can be deployed on the device itself to realize embedded applications, or it can run on a PC or other terminal with a user interface, thereby meeting various hardware environments and usage requirements.

[0133] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wireless or wired transmission; wired transmission methods include optical fiber, twisted pair, coaxial cable, etc.; wireless transmission includes infrared, microwave, etc. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center containing one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0134] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and modules described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0135] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0136] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0137] In addition, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0138] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0139] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0140] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A lightweight on-device inference method for visual language action models, characterized in that, include: S1. Collect computing power usage, power consumption limits, task priorities, and action trajectories and environmental changes over several time periods on the edge device to generate constraint vectors and behavior vectors; S2. Based on the image summarization, instruction representation and sensor quality index input evaluation model, the scene complexity score and input quality score are obtained; S3. Input the constraint vector, behavior vector, scene complexity score, and input quality score into the scheduling model, and output the number of feature segments, the number of cross-modal alignment layers, the action decoding accuracy level, and the safety margin coefficient. The scheduling model scores the scheduling feature vector with the preset level combination according to the configuration table, and selects the level combination with the highest score as the number of feature segments, the number of cross-modal alignment layers, the action decoding accuracy level, and the safety margin coefficient. If no scheduling result is obtained within the preset delay limit, the end device uses the scheduling result recorded in the previous time period, increases the safety margin coefficient level, decreases the feature segment number level and the action decoding accuracy level level, and records the rollback reason and the associated idempotent key in the evidence chain of the operation management library. S4. Based on the number of feature segments, the feature segments are scored for safety relevance during the visual encoding stage. Target region segments and risk region segments are retained first, forming a set of feature segments participating in the visual encoding operation. The edge device marks feature segments overlapping with the target detection box as target region segments based on the target detection results, and marks feature segments overlapping with the risk area as risk region segments based on the preset risk area. The safety relevance score of each feature segment is calculated based on the overlap ratio and the spatial distance to emergency stop events and close-range passing events. The edge device sorts the feature segments according to the safety relevance score based on the feature segment quantity level in the scheduling results, and selects feature segments whose quantity is not greater than the maximum number of feature segments in that level as the set of feature segments participating in the visual encoding operation. When the number of target region segments is less than the maximum number of feature segments in that level, risk region segments are used to supplement it. S5. In the cross-modal alignment stage, compress action-irrelevant instruction segments according to the number of alignment layers. In the action decoding stage, decode safety-critical action components and non-critical action components according to the action decoding accuracy level and safety margin coefficient, and trim the amplitude. S6. At the end of each time period, calculate the stability index based on the current action, historical actions, and environmental changes. When the stability index exceeds the threshold determined by the safety margin coefficient, perform re-inference and switch to a safe action sequence. When the stability index exceeds the stability threshold, the end device writes the stability exceedance mark to the constraint vector in the current time period, increases the safety margin coefficient level, and calls the scheduling model to obtain the updated feature fragment quantity level, cross-modal alignment layer number, and action decoding accuracy level. Under the same observation information, perform re-inference and replace the original action component. When the number of re-inferences reaches the upper limit within the observation window and the stability index still exceeds the stability threshold, the end device selects a safe action sequence from the safe action library that meets the current speed, obstacle distance, and scene category, issues it to the chassis controller, and writes the stability index, stability threshold, number of re-inferences, safe action template identifier, and configuration version number into the operation management library and links them into the evidence chain.

2. The edge-side inference method for a lightweight visual language action model according to claim 1, characterized in that, S1 includes: The edge devices operate in fixed-length time periods driven by a unified timing clock. At the beginning of each time period, the percentage of computing power used, the upper limit of power consumption, the temperature margin, and the current task priority are collected and combined to form a constraint vector. Within an observation window covering a preset number of consecutive time periods, statistical characteristics of the action and environment are calculated based on the changes in the action trajectory and the changes in obstacle distance in each time period, and then combined to form a behavior vector. The runtime management library uses an idempotent key consisting of device identifier, time period number, and task number as an index to associate and store constraint vectors and behavior vectors with the corresponding idempotent keys.

3. The end-side inference method for a lightweight visual language action model according to claim 1, characterized in that, S2 include: The edge device captures images from the front-view camera under a unified time clock, and forms an image summary through resolution compression and channel compression. Obtain instructions from at least one of the voice channel and text channel, and map them into instruction representations according to a preset vocabulary; The sensor quality monitoring unit calculates sensor quality indicators based on image summarization; The edge device aligns the image summary, instruction representation, and sensor quality indicators with the constraint vector and behavior vector according to the unified time synchronization and inputs them into the evaluation model to obtain the scene complexity score and input quality score. The score is then written into the runtime management library using the idempotent key as the index.

4. The edge-side inference method for a lightweight visual language action model according to claim 1, characterized in that, S3 include: The edge device reads the constraint vector, behavior vector, scene complexity score and input quality score associated with the idempotent key from the operation management library based on the idempotent key, and combines the four into a scheduling feature vector according to the preset field order and inputs it into the scheduling model. The scheduling model outputs the number of feature fragments, the number of cross-modal alignment layers, the action decoding accuracy level, and the safety margin coefficient. The output results, along with the scheduling model version number and the configuration table version number, are written to the runtime management library and indexed by an idempotent key.

5. The end-side inference method for a lightweight visual language action model according to claim 1, characterized in that, S5 include: The end-side device parses the instruction representation into action target segments, safety constraint segments, and action-independent instruction segments based on the instruction phrase table; Weights are assigned to action-independent instruction fragments based on behavior vectors and scene complexity scores, and the fragments are compressed into summaries. The summaries, along with action target fragments, safety constraint fragments, and a set of filtered feature fragments, are then input into a cross-modal alignment network whose number of cross-modal alignment layers is determined by scheduling results. The end-side device decodes the joint representation to obtain action components based on the action decoding accuracy level and safety margin coefficient in the scheduling results. The amplitude of safety-critical action components is limited according to the safety margin coefficient, and the action components, together with the configuration table version number, are written into the operation management library and indexed by an idempotent key.

6. The edge-side inference method for a lightweight visual language action model according to claim 1, characterized in that, S6 include: At the end of each time period, the end-side device reads the current safety critical action components, historical safety critical action components, and environmental change records from the operation management library by pressing the idempotent key at the unified time synchronization time. Within the observation window, the variation amplitude and directional consistency of safety-critical action components are statistically analyzed, and the action sequence is aligned with the safety trajectory template to obtain the trajectory deviation. The end-side device combines the amplitude of motion change, directional consistency, and trajectory deviation into a stability index based on preset weights, and retrieves the stability threshold from the stability threshold configuration table based on the current safety margin coefficient, and compares the stability index with the stability threshold.

Images