Low-delay interaction control method and system for unmanned vending cabinet based on edge computing

Abstract

The present application relates to the field of unmanned sales loss prevention control, and provides a low-delay interaction control method and system for an unmanned vending cabinet based on edge computing, which comprises the following steps: obtaining a hand boundary box and a detection confidence based on two-dimensional color image data, combining a multi-order frequency control to generate a dynamic processing frequency signal, if the signal is greater than zero, using the hand boundary box to extract a hand, a product, a risk area node set and a hand motion state set, and constructing a dynamic space-time graph; extracting a graph-level risk feature vector through a space-time attention network according to an interaction intention weight, and outputting a real-time risk prediction value through a regression unit; generating an interaction intervention intensity based on the real-time risk prediction value and a risk safety threshold; determining an interaction intervention level according to the interaction intervention intensity, generating and executing a hardware control instruction, and feeding back an interaction intervention intensity convergence result to determine whether to release the intervention. The present application cooperatively integrates visual perception and adaptive control, and constructs an interaction risk prediction and hierarchical intervention mechanism based on space-time graph modeling.

Description

Technical Field

[0001] This invention relates to the field of loss prevention control in unmanned vending machines, and in particular to a low-latency interactive control method and system for unmanned vending machines based on edge computing. Background Technology

[0002] With the rapid application of open-door unmanned vending machines in the smart retail sector, reliable automated loss prevention has become a key technology to ensure the large-scale operation of their cash-on-delivery system. How to balance product security and customer experience in the complex interactive environment of open shopping, and effectively address the shortcomings of traditional solutions in loss prevention accuracy, real-time response, and user interference, has become a crucial issue that urgently needs to be addressed in the intelligent upgrading of smart retail terminals.

[0003] Chinese patent application CN220208131U discloses a control system for an unmanned vending machine, including a main control module, a secondary control module, and actuators. The main control module has a master transceiver module on a power bus for initiating control commands to dispense goods from the vending channels. The secondary control module has a slave transceiver module on a power bus, electrically connected to the master transceiver module via the power bus. The secondary control modules are located at each vending channel and are used to parse the control commands from the main control module to control the dispensing of goods from the corresponding channels. The actuators are arranged in each vending channel of the unmanned vending machine to respond to the control commands parsed by the corresponding secondary control modules and execute the dispensing action. This method uses a DC carrier communication power bus as both the power supply and communication lines. The secondary control modules control single-layer or single-column vending channels separately, and the integrated power bus provides centralized control. This requires only a small amount of wiring to control multiple layers and columns of vending channels, shortens assembly time, facilitates maintenance, and allows for flexible expansion of vending channels.

[0004] However, current technology still faces many challenges. Traditional visual monitoring systems for open-door vending machines rely heavily on post-event inventory checks for loss prevention, lacking the ability to predict customer interaction intentions in real time. When a customer quickly grabs an item and immediately moves it to a hidden area such as a pocket, backpack, or cuff, the system often struggles to identify this concealment trend within milliseconds. If the system fails to capture and determine the concealment intention before the action is completed, it cannot quickly trigger precise interventions using voice, lights, or other interactive units. Consequently, the loss prevention system can only confirm losses after the fact, but cannot prevent risky behavior from occurring. Once the customer has concealed the item and closed the door, the loss is irreversible, ultimately leading to a high rate of item loss and severely hindering the security and sustainable operation of the cash-on-delivery model. Summary of the Invention

[0005] To achieve the above objectives, this invention provides a low-latency interactive control method for unmanned vending machines based on edge computing, the specific technical solution of which is as follows:

[0006] The system collects multimodal data streams including two-dimensional color image data and three-dimensional point cloud data. Based on the hand bounding box and hand detection confidence obtained in real time from the two-dimensional color image data, it combines multi-order frequency control based on motion sensitivity to determine the hand interaction state and generate a dynamic processing frequency signal. If the dynamic processing frequency signal is greater than zero, it uses the spatial prior information provided by the hand bounding box to extract and fuse the node sets of the hand, goods, risk areas, and hand motion state sets to construct a dynamic spatiotemporal map.

[0007] The dynamic spatiotemporal graph sequence within the time-series sliding window is obtained. Based on the interaction intent weights in the weighted adjacency matrix, a spatiotemporal attention network is used to extract graph-level risk feature vectors in the spatiotemporal dimension. Based on the graph-level risk feature vectors, a lightweight regression unit is used to output real-time risk prediction values.

[0008] Based on real-time risk prediction values ​​and preset risk safety thresholds, the intensity of interactive intervention is generated through multiple processes, including real-time difference calculation, parallel calculation of control components and linear superposition, and saturation function constraints.

[0009] Based on the intensity of the interaction intervention, the level of interaction intervention is determined by a preset piecewise nonlinear mapping function. Hardware control instructions are generated and executed by a control strategy selector according to the level of interaction intervention. The intensity of interaction intervention generated by re-sensing and re-computation is fed back to determine whether convergence has occurred. If convergence reaches a safe state, the intensity of interaction intervention is automatically released.

[0010] Furthermore, the method for constructing the dynamic spatiotemporal graph includes:

[0011] The system acquires the hand bounding box and hand detection confidence in real time from two-dimensional color image data, combines multi-order frequency control based on motion sensitivity to determine the hand interaction state, and generates corresponding dynamic processing frequency signals based on the hand interaction state.

[0012] Time-gated data is obtained by processing a dynamic frequency signal. The three-dimensional point cloud data is then localized by combining the spatial prior information provided by the hand bounding box to generate a three-dimensional spatial volume of interest. The hand skeleton node set and hand motion state set are then extracted from the three-dimensional spatial volume of interest. The hand motion state set includes the velocity vector and acceleration vector of each hand skeleton point.

[0013] Time gating is performed based on dynamic processing frequency signals to obtain two-dimensional color image data and three-dimensional point cloud data. The hand bounding box is expanded into an extended interactive neighborhood using morphological dilation operation. Under the constraints of the extended interactive neighborhood, the product node set and risk area node set are extracted through instance segmentation and back projection calculation.

[0014] Based on the extracted set of hand skeleton nodes, product nodes, risk area nodes, and hand motion state, a global node set is constructed. Using the velocity vector, direction vector between nodes, and three-dimensional Euclidean distance in the hand motion state set, the interaction intention weights of hand and object and hand risk area are quantified to obtain the interaction intention weights. A weighted adjacency matrix is ​​constructed based on the interaction intention weights and combined with the global node set to generate a dynamic spatiotemporal graph.

[0015] Furthermore, the method for determining the hand interaction state based on multi-order frequency control of motion sensitivity includes:

[0016] The multi-order frequency control based on motion sensitivity uses the confidence change rate of the fused hand detection confidence as the judgment index.

[0017] The generation logic of dynamically processed frequency signals is divided into three multi-stage frequency control: high-frequency prediction state, low-frequency prediction state, and sleep state.

[0018] When the confidence level of hand detection is greater than the high control threshold and the rate of change of its confidence level is greater than the motion sensitivity threshold, the system determines that the hand "definitely exists" and is "in a state of rapid movement" and outputs a high processing frequency signal.

[0019] When the confidence level of hand detection is greater than the low control threshold but not greater than the high control threshold, and its confidence level change rate is less than or equal to the motion sensitivity threshold, the system determines that the hand is "present but stationary" or "in a slow motion state" and outputs a low processing frequency signal.

[0020] When the confidence level of hand detection is less than or equal to the low control threshold, the system determines that no intervention is needed, outputs a dynamic processing frequency signal of 0, and remains in a dormant state.

[0021] Furthermore, the hand-object interaction intention weight is the probability of an upcoming interaction between a hand skeleton node from the hand skeleton node set and a product node from the product node set; the hand risk zone interaction intention weight is the probability of an upcoming interaction between a hand skeleton node from the hand skeleton node set and a risk zone node from the risk zone node set.

[0022] Furthermore, the method for outputting the real-time risk prediction value includes:

[0023] The spatiotemporal graph sequence within the time-series sliding window is obtained, and the interaction intent weights in the weighted adjacency matrix are combined. The spatial attention unit is used to aggregate node feature vectors in the spatial dimension to generate a graph-level embedding sequence. Based on the graph-level embedding sequence, the time-series unit is used to extract graph-level risk feature vectors in the time dimension.

[0024] Lightweight regression units are used to perform nonlinear dimensionality reduction and signal mapping on graph-level risk feature vectors, and activation functions are combined to calculate and output real-time risk prediction values.

[0025] Furthermore, the method for generating the graph-level embedding sequence includes:

[0026] The interaction intent weights in the weighted adjacency matrix are introduced as prior knowledge into the calculation process of the spatial attention unit attention to obtain the fusion attention coefficients.

[0027] After obtaining the fusion attention coefficients, the spatiotemporal graph attention network performs weighted aggregation of the neighborhood information of the nodes to generate updated node state vectors, which are then aggregated into graph-level embedding vectors through global average pooling.

[0028] Furthermore, the interaction intent weight is a node and nodes The interaction intention weights between nodes are used to divide the weighted adjacency matrix into virtual intention edges, physical topology edges, and zero-weight edges based on node type.

[0029] The virtual intent edge, if node It belongs to the set of nodes of the hand skeleton, and the nodes If it belongs to the product node set, then the interaction intent weight is the hand-object interaction intent weight; if the node It belongs to the set of nodes of the hand skeleton, and the nodes If a node belongs to a risk area node set, then the interaction intent weight is the same as the interaction intent weight of the risk area.

[0030] The physical topology edge, if node and nodes If all nodes belong to the hand skeleton node set and are physically connected in the skeletal topology of the hand skeleton node set, then the interaction intent weight is set to a fixed non-zero value.

[0031] For all unrelated node pairs, the zero-weight edge has its interaction intent weight set to 0.

[0032] Furthermore, the method for generating the intensity of the interactive intervention includes:

[0033] Based on real-time risk prediction values ​​and preset risk safety thresholds, real-time control errors are generated through real-time difference calculations.

[0034] Based on the real-time control error and the preset control gain parameters, three control components with different characteristics—proportional control component, integral control component, and derivative control component—are calculated and output in parallel; the control gain parameters include proportional gain, integral gain, and derivative gain.

[0035] The decoupled proportional control component, integral control component, and derivative control component are linearly superimposed to obtain a superimposed control value. The superimposed control value is then constrained within a preset physical safety range using a saturation function to generate the interactive intervention intensity.

[0036] Furthermore, the method for automatically reducing the intensity of interactive intervention includes:

[0037] Based on the intensity of the interactive intervention, it is mapped to an interactive intervention level through a preset piecewise nonlinear mapping function. If the intensity of the interactive intervention is less than the low-level threshold, it is determined to be a state of no intervention; if the intensity of the interactive intervention is not less than the low-level threshold and less than the medium-level threshold, it is determined to be soft guidance; if the intensity of the interactive intervention is not less than the medium-level threshold and less than the high-level threshold, it is determined to be a moderate warning; if the intensity of the interactive intervention is greater than or equal to the high-level threshold, it is determined to be strong intervention.

[0038] The interaction intervention level is obtained, and a hardware control instruction set is generated through a control strategy selector. The hardware control instruction set includes visual control instructions, auditory control instructions, and system control instructions. If no intervention is determined, the visual control instructions, auditory control instructions, and system control instructions are all set to "no operation" instructions; if soft guidance is determined, only the visual control instructions are activated; if moderate warning is determined, only the auditory control instructions are activated; if strong intervention is determined, the visual control instructions, auditory control instructions, and system control instructions are activated in parallel.

[0039] Based on the real-time risk prediction value generated by resenting after the execution of the hardware control instruction set, combined with the interactive intervention intensity generated by recalculation, it is determined whether the interactive intervention intensity converges to a safe state. If it converges, the hardware control instruction set is automatically released.

[0040] The edge computing-based low-latency interactive control system for unmanned vending machines is used to implement the aforementioned edge computing-based low-latency interactive control method for unmanned vending machines. The system includes a front-end perception module, a risk prediction module, a risk control module, and a graded intervention module.

[0041] The front-end perception module is used to collect multimodal data streams including two-dimensional color image data and three-dimensional point cloud data. Based on the hand bounding box and hand detection confidence obtained in real time from the two-dimensional color image data, it combines multi-order frequency control based on motion sensitivity to judge the hand interaction state and generate a dynamic processing frequency signal. If the dynamic processing frequency signal is greater than zero, it uses the spatial prior information provided by the hand bounding box to extract and fuse the node set of the hand, the product, the risk area, and the hand motion state set to construct a dynamic spatiotemporal map.

[0042] The risk prediction module is used to acquire the dynamic spatiotemporal graph sequence within the time-series sliding window, extract graph-level risk feature vectors in the spatiotemporal dimension through a spatiotemporal attention network based on the interaction intent weights in the weighted adjacency matrix, and output real-time risk prediction values ​​based on the graph-level risk feature vectors through a lightweight regression unit.

[0043] The risk control module generates the intensity of interactive intervention based on real-time risk prediction values ​​and preset risk safety thresholds by performing multiple processes, including real-time difference calculation, parallel calculation of control components, linear superposition, and saturation function constraints.

[0044] The graded intervention module is used to determine the level of interaction intervention based on the intensity of the interaction intervention through a preset piecewise nonlinear mapping function, generate and execute hardware control instructions through a control strategy selector according to the level of the interaction intervention, and feed back the intensity of the interaction intervention generated by re-sensing and re-computation to determine whether convergence has occurred. If convergence reaches a safe state, the intensity of the interaction intervention is automatically released.

[0045] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0046] This invention integrates the instantaneous kinematic features of the hand skeleton with the spatial relationship between the hand, the goods, and the risk area to calculate and construct a dynamic spatiotemporal map carrying the weight of the interaction intent. This improves the problem of traditional visual solutions relying solely on static posture or spatial distance for delayed recognition, enabling quantitative prediction of high-risk intents such as "taking" or "hiding" within milliseconds before the interaction is completed.

[0047] This invention uses interactive intent weights as prior knowledge and forces their fusion with node features in a spatiotemporal graph attention network. This allows the system to analyze not only isolated kinematic features, such as speed, but also the intent of the motion, i.e., whether it is pointing to a product or a risk area. This improves the problem of high false alarm interference caused by the inability of traditional vision solutions to distinguish between "normal selection" and "high-risk grabbing".

[0048] This invention decouples the real-time control error predicted by AI into proportional control components, integral control components, and derivative control components. It utilizes the feedforward response of the derivative control component on the changing trend and the cumulative characteristics of the integral control component on historical errors to improve the problem that traditional control cannot respond to the "rapid hiding" trend and avoid erroneous intervention in low-risk behaviors such as "normal browsing".

[0049] This invention compares continuous interactive intervention intensity with multi-level grading thresholds and maps low-risk interactive intervention intensity to no-intervention level or soft-guided level, avoiding triggering intrusive intervention commands for low-risk behaviors such as "normal browsing" or "hesitant selection" of customers, and improving the problem of traditional visual solutions frequently interfering with the normal user shopping experience due to high false alarms. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 This is a flowchart illustrating the principle of the low-latency interactive control method for unmanned vending machines based on edge computing of the present invention.

[0052] Figure 2 This is a functional block diagram of the edge computing-based low-latency interactive control system for unmanned vending machines of the present invention. Detailed Implementation

[0053] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] Example 1

[0055] Please see Figure 1 As shown, this embodiment provides a low-latency interactive control method for unmanned vending machines based on edge computing, including:

[0056] S1000, acquires data including two-dimensional color image data and 3D point cloud data Multimodal data stream Based on two-dimensional color image data Real-time acquisition of hand bounding box and confidence level of hand detection By combining multi-order frequency control based on motion sensitivity to determine the hand interaction state, dynamic processing frequency signals are generated. If the dynamic processing frequency signal If the value is greater than zero, then use the hand bounding box. Based on the provided spatial prior information, node sets of hands, goods, and risk areas, as well as hand movement state sets, are extracted and fused to construct a dynamic spatiotemporal graph. .

[0057] Specifically, this step utilizes the near-end processing capabilities of edge computing nodes to process multimodal data streams collected by multiple 3D depth cameras inside the container. The process involves real-time extraction and construction of a lightweight, dynamic spatiotemporal graph with interactive intent weights. This provides decision-making basis for subsequent steps, including kinematic features and user intent features, and realizes the transformation of core data preprocessing from lag recognition to low-latency prediction.

[0058] The multimodal data stream Acquired by a 3D depth camera module triggered by hardware synchronization. At any given time. The camera module outputs two types of data simultaneously: one is two-dimensional color image data. First, it is used to record the color and texture information of the scene; second, it is used in conjunction with the two-dimensional color image data. The depth data, corresponding to each pixel in spatial location, is used to describe the distance information of each pixel in the scene from the camera. After spatial coordinate calculation and geometric reconstruction, the depth data is converted into 3D point cloud data. Therefore, this multimodal data stream Logically includes two-dimensional color image data and 3D point cloud data The two maintain a precise correspondence in time and space.

[0059] Further, step S1000 includes:

[0060] Step S1100: Real-time acquisition of two-dimensional color image data Middle hand bounding box and confidence level of hand detection By combining multi-order frequency control based on motion sensitivity to determine the hand interaction state, a corresponding dynamic processing frequency signal is generated based on the hand interaction state. .

[0061] Specifically, this step, serving as the front-end adaptive controller of the entire hierarchical sensing system, aims to utilize the low-power cores of edge nodes to process the acquired, passive two-dimensional color image data. Real-time interpretation into active, dynamic processing of frequency signals Spatial constraint priors, namely the hand bounding box This enables on-demand dynamic allocation of computing resources, ensuring that edge nodes prioritize the allocation of computing power to interactions with high-risk intentions.

[0062] In the specific implementation process, the low-power core of the Neural Processing Unit (NPU) in the edge computing unit continuously processes the acquired two-dimensional color image data. Lightweight processing is performed at high frame rates. This processing is achieved through an ultra-lightweight 2D hand detector with structural pruning and INT8 quantization, such as YOLOv5-Nano-Pruned.

[0063] The two-dimensional hand detector outputs two key parameters in real time: location within the predefined shelf interaction area. inner hand bounding box and the corresponding hand detection confidence level .in, This represents a predefined shelf interaction area, which is a pre-calibrated two-dimensional image area that corresponds to the physical shelf space where customers can interact in the camera coordinate system. Indicates time The detected hand bounding box represents the spatial coordinates of the hand on the two-dimensional image plane; Indicates time By hand bounding box Limited to a predefined shelf interaction area Confidence level of hand detection within the body.

[0064] To address the issue of confusion between normal customer browsing behavior and potentially high-risk actions, such as rapid grabbing or concealment, this step introduces multi-level frequency control based on motion sensitivity. This control logic not only relies on hand detection confidence... The existence information reflected is creatively integrated with hand detection confidence. confidence level change rate As a motion sensitivity indicator, a multi-order dynamic processing frequency signal is generated. Among them, the rate of change of confidence level It is calculated by temporal difference of the sequence of hand detection confidence, and is used to quantify the instantaneousness or intensity of hand movement, i.e., the motion.

[0065] The dynamic processing frequency signal This is used to adjust the computing frequency at the edge under different interaction states, enabling adaptive resource allocation and response optimization. Its generation logic is divided into three distinct multi-level frequency controls, as follows:

[0066] First, high-frequency prediction state When hand detection confidence level Greater than the high control threshold And its confidence level change rate Greater than the motion sensitivity threshold If the hand is clearly present and in rapid motion, the system determines that it may have a high-risk intention to "take" or "hide." At this point, the system immediately outputs the maximum processing frequency, i.e., a high-processing-frequency signal. It initiates millisecond-level fine-grained 3D tracking.

[0067] Second, low-frequency prediction state When hand detection confidence level Greater than the low control threshold But not greater than the high control threshold And its confidence level change rate Less than or equal to the motion sensitivity threshold If the hand is "present but stationary" or "in a slow movement state," the system determines it to be a low-risk "browsing" or "hovering" behavior. In this case, the system only outputs a low-frequency signal. This is to maintain low-power tracking and avoid high false alarms.

[0068] Third, dormant state: under all other conditions, i.e., hand detection confidence level. Less than or equal to the low control threshold If, for example, no hand is detected, the system determines that no intervention is needed and outputs a dynamically processed frequency signal. Setting it to 0 keeps subsequent high-power steps in sleep mode, achieving low-power no-load operation.

[0069] For example, when a customer's hand quickly reaches for a target product in front of the shelf, i.e., hand detection confidence. Greater than the high control threshold And the rate of change of confidence level Greater than the motion sensitivity threshold If the system determines that the action has a clear intention to pick up an object, it will immediately trigger a high-frequency processing signal. This initiates subsequent steps for high-frequency three-dimensional prediction to address lag. When a customer's hands only move slowly in front of the shelf or hesitate while selecting items, the confidence level of hand detection is determined. Less than the low control threshold But not greater than the high control threshold And the rate of change of confidence level Less than or equal to the motion sensitivity threshold The system only triggers low-frequency signals to maintain a low-power observation state without issuing false alarms, so as to avoid false alarms or interference with normal shopping behavior.

[0070] Step S1200: Based on the dynamic processing frequency signal Perform time gating to obtain 3D point cloud data Combined with hand bounding box The provided spatial prior information for the three-dimensional point cloud data Localization is performed to generate a three-dimensional volume of interest. and from the three-dimensional space focus volume Extracting the hand skeleton node set and hand movement state set .

[0071] Specifically, this step aims to process the frequency signal output in step S1100. Time-gating is performed to enable on-demand activation, and the hand bounding box generated in step S1100 is utilized. The provided spatial prior information constrains the data, enabling computationally efficient processing of the acquired 3D point cloud data with extremely low computational load. Extracting hand skeleton node set with low to medium latency and hand movement state set .

[0072] In the specific implementation process, this step is based on the dynamic processing frequency signal. Time gating is performed only when the dynamic processing frequency signal The system only starts processing multimodal data streams when the value is greater than 0. Capturing 3D point cloud data And according to the dynamic processing frequency signal The specified frequency, such as a high-processing frequency signal. or low-frequency signals Perform subsequent calculations.

[0073] To achieve millisecond-level low-latency processing, this step uses spatial constraints to avoid processing the global 3D point cloud data. Instead of performing expensive calculations, it utilizes a hand-based bounding box. The provided spatial prior information is back-projected through the camera intrinsic matrix of a 3D depth camera, reducing the computational complexity of 3D pose estimation from the global point cloud, i.e., 3D point cloud data. This reduces the focus to a local point cloud, i.e., a dynamic 3D volume of interest. This reduces the computational load by several orders of magnitude, resolving the contradiction between limited edge computing power and millisecond-level low-latency prediction.

[0074] A volume of interest in a three-dimensional space containing only a few thousand points Run a lightweight 3D hand pose estimation algorithm, such as the simplified MediaPipe-3D, to extract... Key skeletal points of the hand, such as .Will The key skeletal points of the hand together constitute the hand skeletal point node set. .in, express The set of hand bone nodes at any given time is a mathematical set whose elements are... The three-dimensional coordinate vectors of each hand bone point are used to describe The static spatial posture of the hand at any given moment; This represents a time index, indicating a momentary state. express The first moment The three-dimensional coordinate vectors of individual hand bone points are used to accurately describe the three-dimensional spatial position of a single bone point, such as the fingertip, palm, or knuckle. Indicates the first Three-dimensional coordinate vector of each hand bone point Belongs to three-dimensional Euclidean space The three-dimensional coordinate vector is specified. It is an ordered tuple of three real numbers, ensuring that the space it describes is three-dimensional; This represents the index of a hand bone point, with values ​​ranging from 1 to the total number of hand bone points. Used to uniquely identify each key skeletal point of the hand, for example It may represent the wrist. It may represent the tip of the index finger; This represents the total number of bone points in the hand.

[0075] To address the hysteresis control issue, this step not only outputs a static set of hand skeleton nodes. It also creatively outputs dynamic sets of hand movement states. This set of hand movement states By adding dynamic information to the static posture, the kinematic state of the hand is captured by calculating the instantaneous velocity and acceleration of each hand skeletal point. express The set of hand kinematic states at time t is a mathematical set whose elements are... The motion characteristics of each hand skeletal point, namely the velocity and acceleration vector pairs, are used to describe... The dynamic movement trends of the hands at all times; express Time of the first The velocity vector of each hand bone point is a three-dimensional coordinate vector. Regarding time The first derivative is used to quantify the hand bone point at... The instantaneous direction and speed of motion at any given moment; express Time of the first The acceleration vector of each hand bone point is a three-dimensional coordinate vector. Regarding time The second derivative or velocity vector Regarding time The first derivative is used to quantify the hand bone point at... The changing trend of the motion state at any given moment, i.e. whether it is accelerating, decelerating or moving at a constant speed.

[0076] Step S1300, based on dynamic processing frequency signal Perform time gating to obtain two-dimensional color image data and 3D point cloud data Using morphological dilation operations to expand the bounding box of the hand Expand to extended interactive neighborhood and in the extended interaction neighborhood Under the constraints, the product node set is extracted through instance segmentation and back projection calculation. and risk area node set .

[0077] Specifically, this step is executed in parallel with step S1200, and aims to process the dynamic frequency signal output in step S1100. Combined with the hand bounding box generated in step S1100 The provided spatial prior information, with extremely low computational load, is derived from the acquired two-dimensional color image data. and 3D point cloud data In the process, the target that the hand is about to interact with, namely the set of product nodes, is extracted with low latency. and risk area node set .

[0078] In the specific implementation process, this step is started in parallel with step S1200, and is also based on the dynamic processing frequency signal. Time gating is performed only when the frequency signal is being dynamically processed. The system only receives data from multimodal data streams when the value is greater than 0. Capture 2D color image data and 3D point cloud data And according to the dynamic processing frequency signal The specified frequency, such as a high-processing frequency signal. or low-frequency signals Perform subsequent calculations.

[0079] To ensure low-latency parallel execution with step S1200, this step does not process the global two-dimensional color image data. Instead of performing instance segmentation, it is based on the hand bounding box. A dynamic, expanding interactive neighborhood is generated through morphological dilation (MD). This morphological dilation operation enables the system to segment not only the bounding box of the hand. The hand can better define the boundary of the hand by examining objects that have already been touched. To address the issue of delayed response, the system can predictively control objects that the external hand is about to touch.

[0080] In the generated extended interactive neighborhood Within a constrained local image region, run a lightweight instance segmentation model, such as a lightweight version of YOLACT or Mask R-CNN, and output the instance segmentation result located within that local image region. Product examples and Two-dimensional mask for high-risk areas.

[0081] for This step involves extracting the centroid position of a product instance from its 2D mask and querying the 3D point cloud data using camera intrinsic parameters. The depth value corresponding to the centroid position is used to calculate its three-dimensional centroid coordinate vector through back projection. Construct a set of product nodes .in, express The set of product nodes at any given time is a mathematical set whose elements are The three-dimensional centroid coordinate vector of each product; express The first moment The three-dimensional centroid coordinate vector of each product is used to accurately describe the spatial location of a single product instance; This represents the index of the product node, with a value ranging from 1 to the total number of products. Used to uniquely identify the extended interaction neighborhood Each product instance detected within; express Constantly expanding the neighborhood of interaction The total number of items detected within each time period, its value at each time period. These can all change dynamically, depending on the hand's position and the distribution of nearby goods.

[0082] Similarly, for For each high-risk area, a two-dimensional mask is used. The system employs the same centroid extraction and back-projection methods to calculate its three-dimensional coordinate vector. Construct a risk area node set .in, express The set of risk zone nodes at time t is a mathematical set whose elements are... Three-dimensional coordinate vectors of high-risk areas, such as backpack openings, pockets, and cuffs; express The first moment The three-dimensional coordinate vectors of each risk area are used to describe the three-dimensional spatial location of a single high-risk area. The index of the risk area node, whose value ranges from 1 to the total number of risk areas. Used to uniquely identify the extended interaction neighborhood Each high-risk area instance detected within; express Constantly expanding the neighborhood of interaction The total number of risk areas detected within the area, its value at each time step. All of these can change dynamically, depending on the customer's posture and belongings.

[0083] Step S1400, based on the extracted hand skeleton node set Product node set Risk area node set and hand movement state set Build a global node set Using hand movement state set The velocity vector, the direction vector between nodes, and the three-dimensional Euclidean distance are quantized into hand-object interaction intention weights. Weight of interaction intent in the risk zone The interaction intent weights are obtained, and a weighted adjacency matrix is ​​constructed based on these interaction intent weights. Combined with global node set To generate dynamic spatiotemporal graphs .

[0084] Specifically, this step aims to extract the hand skeleton node set from step S1200. and the product node set extracted in step S1300 and risk area node set Combined with the hand motion state set output in step S1200 Actively integrate and elevate the dimensions to construct a weighted graph that carries predictive intent, namely a dynamic spatiotemporal graph. .

[0085] In the specific implementation process, this step is in Time-based aggregation of hand skeleton node sets Hand movement status set Product node set and risk area node set Construct a complete global node set .in, express The global node set at time step is a mathematical set whose elements are The three-dimensional coordinate nodes of all entities of interest in the scene at any given moment, namely the hand, the product, and the risk area; This represents the union operation of sets.

[0086] Next, construct a weighted adjacency matrix. , used to describe the global node set The dynamic interaction relationships between nodes. The innovation of this step lies in the fact that the weights of the virtual intent edges between nodes are no longer simple distances or binary connections, but rather incorporate the hand motion state set. The quantification of interaction intent can be divided into the interaction intent weight of the hand object. Weight of interaction intent in the risk zone By weighting the hand-object interaction intent Weight of interaction intent in the risk zone The two types of weights are integrated into the weighted adjacency matrix. This allows the system to reflect the strength of behavioral intentions at the topological level, achieving a higher-dimensional modeling from geometric associations to semantic intentions.

[0087] The weight of the hand-object interaction intention It comes from the hand skeleton node set. The Each hand bone node With the product node set The Product Nodes The probability of an upcoming interaction between two entities. The specific calculation formula is as follows:

[0088] ;

[0089] in, The Sigmoid activation function is a non-linear function used to normalize the intention value of any real number to the interval (0,1), making it a weight value of a class probability. express Time of the first The velocity vectors of each hand skeleton point are directly derived from the hand motion state set. ; express From the moment Each hand bone node Pointing to the Product Nodes The unit direction vector is a vector with a magnitude of 1; express Time of the first Each hand bone node and the Product Nodes The three-dimensional Euclidean distance between them; This represents a tiny positive number, a preset, near-zero positive number used to ensure that the denominator is not zero in the extreme case of hand contact with the product, thus guaranteeing the stability of the calculated value.

[0090] Similarly, this step uses the interaction intent weighting with the object. The exact same logic applies to calculating the node set from the hand skeleton. The Each hand bone node With nodes from risk areas The Risk area nodes Interaction intention weight between hand risk zones This is used to quantify the intent to conceal.

[0091] Finally, this step integrates the aforementioned global node set. and weighted adjacency matrix Construct and output a dynamic spatiotemporal graph carrying interaction intent weights. .

[0092] S2000, acquire the dynamic spatiotemporal graph sequence within the time-series sliding window. According to the weighted adjacency matrix Interaction Intent Weight Graph-level risk feature vectors are extracted in the spatiotemporal dimension using a spatiotemporal attention network. And based on the graph-level risk feature vector Output real-time risk prediction values ​​through a lightweight regression unit. .

[0093] Specifically, this step aims to transform the high-dimensional dynamic spatiotemporal diagram output by the S1400... Risk trend analysis is performed using a time-series graphical neural network, which reduces the dimensionality and maps it to a single, continuous process variable, i.e., the real-time risk prediction value. This bridges the semantic gap between high-dimensional perception and real-time control.

[0094] Further, step S2000 includes:

[0095] Step S2100: Obtain the spatiotemporal graph sequence within the time-series sliding window. Combined with weighted adjacency matrix Interaction Intent Weight Spatial attention units are used to aggregate node feature vectors in the spatial dimension to generate graph-level embedding sequences. Based on the graph-level embedding sequence, graph-level risk feature vectors are extracted in the time dimension using temporal units. .

[0096] Specifically, this step aims to utilize the dynamic spatiotemporal graph output in step S1400. A spatio-temporal graph attention network (ST-GAT) integrating prior knowledge is constructed to aggregate node features in both spatial and temporal dimensions, extracting a high-dimensional feature vector representing the evolution trend of risk behavior in the scene, i.e., a graph-level risk feature vector. .

[0097] In the specific implementation process, this step obtains the dynamic spatiotemporal graph of the current frame. And its historical evolution context. A timing sliding window is set in the cache of the edge computing unit to store a fixed-size [data / data / etc.]. The spatiotemporal graph data of the frame. This sliding window uses a First-In-First-Out (FIFO) queue. Current frame dynamic spatiotemporal graph at time step The earliest data frame when pushed into the buffer Discarded. This step reads a complete, containing... from the cache. Frame sequence of consecutive time graphs It is then fed into the spatiotemporal graph attention network ST-GAT, which is deployed on the neural network processing unit (NPU) in the edge computing unit.

[0098] The innovation of the Spatiotemporal Graph Attention Network (ST-GAT) lies in its spatial attention unit. At that moment, the spatial attention unit no longer learns the global node set from scratch. Instead of focusing on the importance of any two nodes, it innovatively uses a weighted adjacency matrix. Interaction Intent Weight Introduced as prior knowledge into the attention calculation process, the fused attention coefficient is obtained. This achieves a synergistic integration of structural priors and dynamic learning. The specific process formula is as follows:

[0099] ;

[0100] ;

[0101] in, and Represents the global node set The general node index in the text is used to avoid ambiguity with the special indexes defined in step S1400, namely the indexes of hand bone points. Index of product nodes Index of risk area nodes ; and They represent Time Node and nodes The input node feature vectors, whose initial values ​​are the global node set. The three-dimensional coordinate vector of the node; This indicates a feature concatenation operation, used to... , and The three features are connected in a dimension to form a higher-dimensional fused feature vector; The attention head function is a learnable neural network layer used to calculate the unnormalized attention score of the fused feature vector after feature concatenation. express Time Node and nodes The fusion attention score between them; Indicates in The dimensional normalization function is an activation function used to fuse attention scores. Transform into a set of fusion attention coefficients that sum to 1 .

[0102] The interaction intent weight express Time Node and nodes The interaction intent weights between them are derived directly from the weighted adjacency matrix. In Location extraction. This weighted adjacency matrix. It carries virtual intent edges and physical topology edges. Based on node type, it is divided into four cases, as follows:

[0103] Scenario 1: Virtual Intent Edge. If the node Belongs to the global node set Hand skeleton node set That is, the index of the hand bones. And nodes Belongs to the global node set Product node set in That is, the index of the product node. Then the interaction intent weight Weights of hand-object interaction intent in step S1400 .

[0104] Scenario 2, Virtual Intent Edge. If node Belongs to the global node set Hand skeleton node set That is, the index of the hand bones. And nodes Belongs to the global node set risk area node set That is, the index of the risk area node. Then the interaction intent weight Weight of the hand risk area interaction intent in step S1400 .

[0105] Scenario 3: Physical topology edge. If node and nodes All belong to the global node set Hand skeleton node set Furthermore, if they are physically connected in the skeletal topology extracted in step S1200, such as "fingertip" and "second knuckle," then the interaction intent weight... It is set to a fixed non-zero value.

[0106] Case 4, Zero-weighted edges. For all other unrelated node pairs, such as "product-product" or "product-risk zone", their interaction intent weight is zero. Setting it to 0 indicates that the spatiotemporal graph attention network ST-GAT should ignore these zero-weight edges during spatial aggregation.

[0107] In obtaining the fusion attention coefficient Subsequently, the spatiotemporal graph attention network ST-GAT is used to process the nodes. The neighborhood information is weighted and aggregated to generate an updated node state vector. And through Global Average Pooling (GAP) operations, the pooling process... All updated node state vectors at time step Aggregation into graph-level embedding vectors .

[0108] The spatiotemporal graph attention network ST-GAT is used for input... Frame timing sequence Each frame in the process undergoes the spatial aggregation process described above, calculating and generating corresponding graph-level embedding vectors, which together form a graph-level embedding sequence. This graph-level embedding sequence The temporal units, such as gated recurrent units (GRUs), input to the spatiotemporal graph attention network ST-GAT are used to capture the evolutionary trend of intent over time, thereby outputting the final graph-level risk feature vector. .

[0109] Step S2200: Utilize lightweight regression units to analyze the graph-level risk feature vector. Nonlinear dimensionality reduction and signal mapping are performed, and activation functions are used to calculate and output real-time risk prediction values. .

[0110] Specifically, this step aims to transform the high-dimensional and abstract graph-level risk feature vector output from step S2100... Through a lightweight regression head, the dimensionality is reduced and transformed into a real-time, continuous, and normalized real-time risk prediction value that can be processed in real time. This enables dynamic quantitative assessment of customer behavior status within milliseconds.

[0111] In the specific implementation process, the obtained graph-level risk feature vector The input is fed into a lightweight regression unit deployed on an edge NPU. This regression unit typically consists of a multilayer perceptron (MLP) or a single-layer fully connected layer (FC), primarily processing graph-level risk feature vectors. Nonlinear dimensionality reduction and signal mapping are performed, and finally, a scalar signal, namely the real-time risk prediction value, is output through a sigmoid activation function. .

[0112] For example, when a customer's hand movements rapidly approach a specific high-risk area, such as the lower edge of a cabinet door or an obstructed area, and their movement trajectory contradicts the direction of the merchandise node, the graph-level risk feature vector extracted in step S2100... Significant anomalies will be observed in the corresponding dimensions. After this mapping step, the output risk prediction value will be... The confidence level will rapidly rise to near 1 in the high confidence interval. At this point, the edge control unit can immediately trigger subsequent high-frequency intervention commands, such as voice prompts or local video enhancement, thereby completing risk control before the customer's action is completed.

[0113] S3000, based on real-time risk prediction values and preset risk and safety thresholds By performing multiple processes, including real-time subtraction, parallel computation of control components, linear superposition, and saturation function constraints, the intensity of interactive intervention is generated. .

[0114] Specifically, this step aims to transform the real-time risk prediction value output from step S2200... Compared with preset control settings and risk safety thresholds Perform closed-loop control calculations to generate a smooth, continuous interactive intervention intensity with predictive control characteristics. This enables the conversion of AI predictions into real-time control commands within milliseconds.

[0115] Further, step S3000 includes:

[0116] Step S3100, based on real-time risk prediction values and preset risk and safety thresholds Real-time control error is generated through real-time difference calculation. .

[0117] Specifically, this step aims to achieve control within each millisecond-level control cycle. The real-time risk prediction value output by the real-time quantification step S2200 and the system's preset, tolerable risk safety threshold The instantaneous deviation between them, i.e., real-time control error .

[0118] In the specific implementation process, at time This step obtains the real-time risk prediction value output from the previous step. It also reads the control settings, i.e., the risk and safety threshold, from the parameter memory of the edge node. By performing a real-time subtraction operation on the edge computing unit, the real-time risk prediction value is obtained. Subtract the risk safety threshold Calculate the time Real-time control error This real-time control error It is a continuous scalar, and a positive value indicates the current real-time risk prediction. The risk has exceeded the system's tolerance threshold. .

[0119] For example, when a customer's hands are normally browsing products in front of a shelf, the real-time risk prediction value... The risk and safety threshold set by the system is approximately 0.05. The value is 0.1. This step calculates the real-time control error. Due to real-time control errors A negative value indicates that the current state is in the "safe range" and no false alarms are triggered, effectively eliminating the high false alarm problem common in traditional two-dimensional vision solutions and achieving stable control with low power consumption and low intervention.

[0120] When a customer's hands quickly move towards concealed areas such as pockets or body, the real-time risk prediction value... The risk threshold is approximately 0.9, which is the system's set risk and safety threshold. It remains at 0.1. This step calculates the real-time control error. This significant positive deviation indicates that the system has identified a "high-risk behavioral trend." An intervention signal is generated before the action is completed, triggering a millisecond-level response. This causes the interactive unit to issue a prompt voice or light signal in advance, preventing potential hiding behavior from occurring.

[0121] Through the aforementioned collaborative mechanism, an integrated "prediction-response" control logic is achieved, enabling the system to not only possess traditional automatic feedback adjustment capabilities but also feedforward intervention capabilities for future behavioral trends. This allows the system to proactively suppress risk lag and significantly reduce high false alarm rates when dealing with both normal and abnormal behaviors, achieving a comprehensive effect far exceeding that of single visual recognition or traditional control strategies.

[0122] Step S3200, based on real-time control error The proportional control component is calculated in parallel with the preset control gain parameters and outputs the proportional control component. Integral control components and differential control components Three control components with different characteristics; the control gain parameter includes proportional gain. Integral gain and differential gain .

[0123] Specifically, this step aims to process the real-time control error output from step S3100 at the edge nodes. In the time domain, it is decoupled into three control components with different dynamic characteristics, namely proportional control components. Integral control components and differential control components It enables tiered responses to current risks, continuously accumulating risks, and sudden trend risks, and works in conjunction with AI prediction results.

[0124] In the specific implementation process, this step occurs in each control cycle of the edge computing unit. Internally, based on real-time control error Parallel computing of proportional control components Integral control components and differential control components Three control components.

[0125] The proportional control component Used for the current Real-time control error at any moment To produce an instantaneous, proportional response, handle short-term and sudden risk deviations, and ensure the system has rapid response capabilities. The specific process formula is as follows:

[0126] ;

[0127] in, Representing proportional gain, it is a preset, online-tunable engineering parameter scalar used to define the proportional control component. For real-time control error The response sensitivity.

[0128] The integral control component By accumulating from 0 to All historical errors at any given moment are used to address low-risk, long-lasting behaviors, such as a customer slowly concealing merchandise. The specific process formula is as follows:

[0129] ;

[0130] in, Indicates the previous moment The integral control component is at the current moment. Integral control component The calculation results from the previous control cycle are used at time [time]. Implement recursive integration or time-domain accumulation; The integral gain is a preset, online-tunable engineering parameter scalar used to define the real-time control error. The accumulated speed; The control cycle is a very short time interval.

[0131] The differential control component By calculating real-time control error The rate of change of [the value] is used for predictive control to address behaviors that escalate rapidly in risk, such as customers quickly hiding goods, thus achieving feedforward control of high-risk trends. The specific process formula is as follows:

[0132] ;

[0133] in, Representing the differential gain, it is a preset, tunable engineering parameter scalar used to define the differential control component. For real-time control error Response sensitivity to rate of change; Indicates the previous moment Control error.

[0134] Step S3300, decouple the proportional control components Integral control components and differential control components The superimposed control value is obtained through linear superposition. The superimposed control value is then constrained within a preset physical safety range using a saturation function to generate the interactive intervention intensity. .

[0135] Specifically, this step aims to decouple the proportional control component from step S3200. Integral control components and differential control components The three control components are linearly superimposed and subjected to saturation constraints to generate a single and continuous interactive intervention intensity. .

[0136] In the specific implementation process, this step decouples the three control components output from the previous steps, namely the proportional control components. Integral control components and differential control components Linear superposition is performed to obtain superimposed control values. These superimposed control values ​​are then constrained within a preset physical safety range using a nonlinear saturation function, generating... Intensity of interactive intervention at any time .

[0137] The saturation function is a nonlinear limiting function, i.e., a limiter, whose inputs include the superimposed control value and upper and lower limits of the physical safety range, used to prevent the superimposed control value from exceeding the safety range; the physical safety range is defined by the lower limit. and upper limit Composition, in which, This represents the minimum intervention intensity, i.e., the lower limit of the superimposed control value variable. It is a preset engineering parameter scalar used to define the lower limit of the saturation function. This represents the maximum intervention intensity, i.e., the upper limit of the superimposed control value variable. It is a preset engineering parameter scalar used to define the upper limit of the saturation function.

[0138] S4000, based on the intensity of interactive intervention The level of interactive intervention is determined by a preset piecewise nonlinear mapping function. Based on the aforementioned level of interaction intervention Hardware control instructions are generated and executed through the control strategy selector. It also provides feedback on the intensity of interactive interventions generated by re-perception and re-computation. To determine whether convergence has occurred, if convergence has reached a safe state, the interaction intervention intensity will be automatically reduced. .

[0139] Specifically, this step aims to increase the intensity of the interactive intervention generated in step S3300. It interprets and maps the commands into discrete, hierarchical interactive control instructions. This enables tiered intervention at different interaction levels, thereby closing the adaptive control closed loop from step S1000 to step S4000.

[0140] Further, step S4000 includes:

[0141] Step S4100, based on the intensity of interactive intervention Mapped to the level of interactive intervention through a preset piecewise nonlinear mapping function. If the intensity of the interactive intervention Less than the low-level threshold If so, it is determined as no intervention. The state; if the intensity of the interactive intervention Not less than the low-level threshold And less than the middle-level threshold If so, it is determined to be a soft boot. If the intensity of the interactive intervention Not less than the middle-level threshold And less than the high-level threshold If so, it is judged as a moderate warning. If the intensity of the interactive intervention Greater than or equal to the high-level threshold If so, it is judged as forceful intervention. .

[0142] Specifically, this step aims to increase the intensity of the interactive intervention generated in step S3300. The data is discretized in real time using a nonlinear mapping function in the edge nodes, and the output is a progressively escalating level of interactive intervention. .

[0143] In the specific implementation process, this step deploys a pre-defined piecewise nonlinear mapping function on the edge nodes to represent the continuous interaction intensity. Millisecond-level real-time mapping to a discrete level of interactive intervention. The level of interactive intervention. It is an enumeration value whose range is a preset set of intervention levels. ,in Indicates no intervention, Indicates soft boot, Indicates a moderate warning and This indicates strong intervention. The specific mapping logic is as follows:

[0144] Logic 1: No intervention Mapping. Determining the intensity of interactive intervention. Is it less than the low-level threshold? If the intensity of the interactive intervention Less than the low-level threshold If the system determines that the current situation is low-risk, such as when the customer is simply browsing or comparing product labels normally in front of the shelf, then this step will adjust the intensity of the interactive intervention. Mapped to no intervention This state allows for the suppression of unnecessary intervention signals at the program control layer, thus resolving the high false alarm problem.

[0145] Logic 2, Soft Boot The logic. If the intensity of interactive intervention... Not less than the low-level threshold The system then determines whether it is less than the intermediate threshold. If the intensity of this interactive intervention Greater than and equal to the low-level threshold And less than the middle-level threshold If this occurs, the system determines that the current risk level is medium or the risk level is accumulating slowly. This step will determine the intensity of the interactive intervention. Mapped to soft boot The state is such that soft intervention is initiated to achieve adaptive intervention control.

[0146] Logic 3, Moderate Warning The logic. If the intensity of interactive intervention... Not less than the middle-level threshold The system then continues to determine whether it is less than the higher-level threshold. If the intensity of this interactive intervention Greater than and equal to the middle classification threshold And less than the high-level threshold When this occurs, it indicates that the risk has been identified, and this step will adjust the intensity of the interactive intervention. Mapped to moderate warning The system is in a state that triggers clear audible and visual warnings or visual cues to achieve an early warning response.

[0147] Logic 4: Strong Intervention The logic. If the intensity of interactive intervention... Greater than or equal to the high-level threshold If the system determines that a high-risk, sudden trend is emerging, such as a customer hiding or taking more items than they need, this step immediately adjusts the intensity of the interaction intervention. Mapped to strong intervention The state triggers a powerful intervention command, such as a delayed door lock or background recording activation, to achieve millisecond-level predictive intervention in predictive control and solve the problem of lag.

[0148] Among them, the low-level threshold Medium-level threshold High-level threshold These are three preset, online-tunable engineering parameter scalars, whose values ​​must meet low-level threshold requirements. Less than the middle-level threshold And the middle-level threshold Less than the high-level threshold The logic is used to define the intensity of interactive intervention. Mapped to different levels of interactive intervention It is used to balance high false alarms and hysteresis.

[0149] Step S4200: Obtain the level of interactive intervention. The hardware control instruction set is generated through the control strategy selector. The hardware control instruction set Including visual control commands Auditory control commands and system control commands If determined to be without intervention Then the visual control command will be executed. Auditory control commands and system control commands All are set to NULL (no-operation) instruction; if determined to be a soft boot. Then only the visual control command is activated. If determined as a moderate warning Then only auditory control commands are activated. If determined to be forceful intervention Then the visual control commands are activated in parallel. Auditory control commands and system control commands .

[0150] Specifically, this step aims to address the level of interactive intervention output in step S4100. It immediately queries and generates a hardware control instruction set corresponding to the level of interactive intervention within milliseconds. It coordinates multiple interactive control units on the edge side, such as AI digital humans, voice units, screens, and door lock controllers, to conduct closed-loop intervention for adaptive control.

[0151] In the specific implementation process, this step uses a control strategy selector on the edge computing node to discretize the interaction intervention levels. Translated into a specific hardware control instruction set The hardware control instruction set It is not a single instruction, but a multi-channel hardware control instruction set, containing multiple hardware control vectors, i.e., vision control instructions. Auditory control commands and system control commands The visual control commands The auditory control commands are used to control the AI ​​digital human screen or lighting strip. The system control commands are used to control the voice unit. Used for advanced procedures such as controlling door locks, making payments, or settling accounts.

[0152] The selection logic of the control strategy selector is as follows:

[0153] When the level of interaction intervention No intervention At that time, the control strategy selector will then switch to no intervention. The state is translated into a fully silent hardware control instruction set. In this control instruction set, all sub-actuator channels, including vision control instructions... Auditory control commands and system control commands All were explicitly assigned the value NULL, the "no operation" instruction. The NULL instruction explicitly indicates the visual control command. Auditory control commands and system control commands Without intervention Remain silent in this state to ensure the tiered nature of the intervention.

[0154] When the level of interaction intervention soft boot At this time, the control strategy selector only activates visual control commands. At the same time, explicitly transmit auditory control commands. and system control commands Set to NULL for "no operation". This soft, non-intrusive intervention addresses the serious damage to the normal user shopping experience caused by high false alarms in known technologies, such as voice warnings, at the actuator level. The visual control instructions... When activated, it does not execute a static instruction, such as blinking, but rather executes a data-driven coordinating function that inversely coordinates the hand skeleton node set output by step S1200. This enables the AI ​​digital human's gaze to adaptively follow the movement of hand skeletal points. The "NULL" instruction is used to explicitly indicate auditory control commands. and system control commands Without intervention Remain silent in this state to ensure the tiered nature of the intervention.

[0155] When the level of interaction intervention Moderate warning At this time, the control strategy selector only activates auditory control commands. Assign it the "moderate voice alarm" command. For example, triggering the voice unit to say "Abnormal interaction detected, please operate correctly," while simultaneously explicitly displaying visual control commands. and system control commands Set to the "no operation" instruction NULL. The "no operation" instruction NULL is used to explicitly indicate the visual control instruction. and system control commands Without intervention Remain silent in this state to ensure the tiered nature of the intervention.

[0156] When the level of interaction intervention For strong intervention At that time, the control strategy selector activates all relevant actuator channels in parallel, triggering not only visual control commands. The "bright light flashing" command FlashLight and the auditory control command "Stern voice alert" command This serves to intimidate users and simultaneously trigger system control commands. Set it to the "ForceReview" instruction. The "FlashLight" instruction is used to activate the vision actuator channel, i.e., the vision control instruction. For example, it can trigger the LED light strip inside the container to flash at a high frequency to achieve a strong visual warning; the "stern voice alarm" command Used to activate the auditory actuator channel, i.e., auditory control commands. For example, it can trigger the voice unit to play a stern warning tone saying "High-risk behavior has been recorded!"; the "ForceReview" command is used to activate the system status control channel, i.e., the system control command. This instruction forces subsequent procedures, such as payment procedures, to change their status at the predictive control level, for example, "disable automatic settlement" and switch to "manual review process".

[0157] Step S4300, based on hardware control instruction set After execution, the real-time risk prediction value generated based on the re-sensing in steps S1000 to S2000 is... Combined with the interactive intervention intensity regenerated in step S3000 Determine the intensity of the interactive intervention. Has the convergence reached a safe state? If convergence occurs, the hardware control instruction set will be automatically released. .

[0158] Specifically, this step aims to describe the hardware control instruction set output by step S4200. As the starting point of the feedback loop, how can the sensor of the edge computing node in step S1000 be coordinated in reverse through the behavioral changes of the controlled object, i.e., the behavioral changes of the customer, to achieve the adaptive loop convergence of the system?

[0159] In the specific implementation process, this step describes the collaborative logic of the adaptive control closed loop formed by steps S1000 to S4000. This closed loop deeply integrates AI perception and control, and its signal flow is generated by the hardware control instruction set output in step S4200. Trigger: This instruction set acts directly on the controlled object in the physical world, namely the customer.

[0160] When hardware control instruction set Once activated, the customer's status changes accordingly. For example, in a moderate warning setting. Next, customers will move their hands away from the risk area nodes. That is, remove it from the pocket or backpack. Next, the multiple 3D depth cameras configured in step S1000 acquire and execute hardware control instruction sets in real time. The subsequent multimodal data stream The system captures changes in the customer's hand posture. These changes in posture result in the hand risk area interaction intent weight output in step S1400. The sharp decline leads to a decrease in the process variable output by step S2000, namely the real-time risk prediction value. Simultaneous descent.

[0161] This real-time risk prediction value It is sent back to step S3100 to calculate the result. Real-time control error within the cycle It is a negative value. This is the real-time control error. The proportional control component is immediately fed into step S3200, causing it to be computed in parallel. and differential control components It returns to zero or becomes negative. Based on this, step S3300 generates... Control variables are generated at all times, namely the intensity of interactive intervention. After being limited by the saturation function, it gradually converges.

[0162] Finally, step S4100 will converge the interaction intervention strength to near zero. Remapping to no intervention Step S4200 automatically releases the hardware control instruction set. The adaptive control loop closes and converges to a safe state.

[0163] For example, in At any given time, the system is in a state of no intervention. Status. In At a certain moment, the edge node predicts that a customer will quickly hide items; at this point, the intensity of the interactive intervention... Instantly rising to 10.0, step S4100 immediately increases this interaction intervention intensity. Mapped to strong intervention .exist At that moment, strong intervention was carried out in step S4200. Corresponding hardware control instruction set This includes triggering visual control commands. FlashLight command, auditory control command "Stern voice alert" command and system control commands The "ForceReview" directive. At any moment, the controlled object is subjected to strong intervention. The deterrent effect prompted them to immediately cease concealing the goods and to collect the hand bone node data. Move back to the predefined shelf interaction area .exist At that moment, step S1000 captures the set of skeletal points on the customer's hand. Based on the spatial location change, step S2000 outputs a real-time risk prediction value. .exist At that moment, step S3100 calculates the real-time control error. At this point, the proportional control component calculated in step S3200... and differential control components It immediately becomes a negative value. The intensity of the interactive intervention output by step S3000. Converging to 0 in milliseconds. Finally, step S4100 determines the level of interactive intervention. Restore to no intervention , The hardware control instruction set of step S4200 is automatically released at all times. The system returns to steady state through closed-loop operation.

[0164] Example 2

[0165] This embodiment, based on Embodiment 1, provides a low-latency interactive control system for unmanned vending machines based on edge computing, such as... Figure 2 As shown, it includes a front-end perception module, a risk prediction module, a risk control module, and a tiered intervention module;

[0166] The front-end sensing module is used to collect data including two-dimensional color image data. and 3D point cloud data Multimodal data stream Based on two-dimensional color image data Real-time acquisition of hand bounding box and confidence level of hand detection By combining multi-order frequency control based on motion sensitivity to determine the hand interaction state, dynamic processing frequency signals are generated. If the dynamic processing frequency signal If the value is greater than zero, then use the hand bounding box. Based on the provided spatial prior information, node sets of hands, goods, and risk areas, as well as hand movement state sets, are extracted and fused to construct a dynamic spatiotemporal graph. .

[0167] The risk prediction module is used to acquire dynamic spatiotemporal graph sequences within a time-series sliding window. According to the weighted adjacency matrix Interaction Intent Weight Graph-level risk feature vectors are extracted in the spatiotemporal dimension using a spatiotemporal attention network. And based on the graph-level risk feature vector Output real-time risk prediction values ​​through a lightweight regression unit. .

[0168] The risk control module is based on real-time risk prediction values. and preset risk and safety thresholds By performing multiple processes, including real-time subtraction, parallel computation of control components, linear superposition, and saturation function constraints, the intensity of interactive intervention is generated. .

[0169] The tiered intervention module is used to determine the intensity of the interactive intervention. The level of interactive intervention is determined by a preset piecewise nonlinear mapping function. Based on the aforementioned level of interaction intervention Hardware control instructions are generated and executed through the control strategy selector. It also provides feedback on the intensity of interactive interventions generated by re-perception and re-computation. To determine whether convergence has occurred, if convergence has reached a safe state, the interaction intervention intensity will be automatically reduced. .

[0170] The parts of the technical solutions provided in the embodiments of this application that are consistent with the implementation principles of corresponding technical solutions in the prior art have not been described in detail to avoid excessive elaboration.

[0171] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the invention. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A low-latency interactive control method for unmanned vending machines based on edge computing, characterized in that, include: The system collects multimodal data streams including two-dimensional color image data and three-dimensional point cloud data. Based on the hand bounding box and hand detection confidence obtained in real time from the two-dimensional color image data, it combines multi-order frequency control based on motion sensitivity to determine the hand interaction state and generate a dynamic processing frequency signal. If the dynamic processing frequency signal is greater than zero, it uses the spatial prior information provided by the hand bounding box to extract and fuse the node sets of the hand, goods, risk areas, and hand motion state sets to construct a dynamic spatiotemporal map. The method for judging hand interaction state based on multi-order frequency control of motion sensitivity includes: the multi-order frequency control based on motion sensitivity uses the confidence change rate of the fused hand detection confidence as the judgment index. The generation logic of dynamically processed frequency signals is divided into three multi-stage frequency control: high-frequency prediction state, low-frequency prediction state, and sleep state. The high-frequency prediction state is when the confidence level of hand detection is greater than the high control threshold and the rate of change of its confidence level is greater than the motion sensitivity threshold, the system determines that the hand "definitely exists" and is "in a rapid motion state", and outputs a high processing frequency signal. The low-frequency prediction state is when the confidence level of hand detection is greater than the low control threshold but not greater than the high control threshold, and the rate of change of its confidence level is less than or equal to the motion sensitivity threshold, the system determines that the hand is "present but stationary" or "in a slow motion state" and outputs a low processing frequency signal. The dormant state is when the confidence level of hand detection is less than or equal to the low control threshold, the system determines that no intervention is needed, outputs a dynamic processing frequency signal of 0, and remains in the dormant state. The dynamic spatiotemporal graph sequence within the time-series sliding window is obtained. Based on the interaction intent weights in the weighted adjacency matrix, a spatiotemporal attention network is used to extract graph-level risk feature vectors in the spatiotemporal dimension. Based on the graph-level risk feature vectors, a lightweight regression unit is used to output real-time risk prediction values. Based on real-time risk prediction values ​​and preset risk safety thresholds, the intensity of interactive intervention is generated through multiple processes, including real-time difference calculation, parallel calculation of control components and linear superposition, and saturation function constraints. The method for generating the intensity of the interactive intervention includes: generating a real-time control error by performing a real-time difference calculation based on the real-time risk prediction value and the preset risk safety threshold; Based on the real-time control error and the preset control gain parameters, three control components with different characteristics—proportional control component, integral control component, and derivative control component—are calculated and output in parallel; the control gain parameters include proportional gain, integral gain, and derivative gain. The decoupled proportional control component, integral control component and derivative control component are linearly superimposed to obtain the superimposed control value. The superimposed control value is constrained within a preset physical safety range using a saturation function to generate the interactive intervention intensity. Based on the intensity of the interaction intervention, the level of interaction intervention is determined by a preset piecewise nonlinear mapping function. Hardware control instructions are generated and executed by a control strategy selector according to the level of interaction intervention. The intensity of interaction intervention generated by re-sensing and re-computation is fed back to determine whether convergence has occurred. If convergence reaches a safe state, the intensity of interaction intervention is automatically released.

2. The low-latency interactive control method for unmanned vending machines based on edge computing according to claim 1, characterized in that, The method for constructing the dynamic spatiotemporal graph includes: The system acquires the hand bounding box and hand detection confidence in real time from two-dimensional color image data, combines multi-order frequency control based on motion sensitivity to determine the hand interaction state, and generates corresponding dynamic processing frequency signals based on the hand interaction state. Time-gated data is obtained by processing a dynamic frequency signal. The three-dimensional point cloud data is then localized by combining the spatial prior information provided by the hand bounding box to generate a three-dimensional spatial volume of interest. The hand skeleton node set and hand motion state set are then extracted from the three-dimensional spatial volume of interest. The hand motion state set includes the velocity vector and acceleration vector of each hand skeleton point. Time gating is performed based on dynamic processing frequency signals to obtain two-dimensional color image data and three-dimensional point cloud data. The hand bounding box is expanded into an extended interactive neighborhood using morphological dilation operation. Under the constraints of the extended interactive neighborhood, the product node set and risk area node set are extracted through instance segmentation and back projection calculation. Based on the extracted set of hand skeleton nodes, product nodes, risk area nodes, and hand motion state, a global node set is constructed. Using the velocity vector, direction vector between nodes, and three-dimensional Euclidean distance in the hand motion state set, the interaction intention weights of hand and object and hand risk area are quantified to obtain the interaction intention weights. A weighted adjacency matrix is ​​constructed based on the interaction intention weights and combined with the global node set to generate a dynamic spatiotemporal graph.

3. The low-latency interactive control method for unmanned vending machines based on edge computing according to claim 2, characterized in that, The hand-object interaction intent weight is the probability of an upcoming interaction between a hand skeleton node from the hand skeleton node set and a product node from the product node set; the hand risk zone interaction intent weight is the probability of an upcoming interaction between a hand skeleton node from the hand skeleton node set and a risk zone node from the risk zone node set.

4. The low-latency interactive control method for unmanned vending machines based on edge computing according to claim 1, characterized in that, The method for outputting the real-time risk prediction value includes: The spatiotemporal graph sequence within the time-series sliding window is obtained, and the interaction intent weights in the weighted adjacency matrix are combined. The spatial attention unit is used to aggregate node feature vectors in the spatial dimension to generate a graph-level embedding sequence. Based on the graph-level embedding sequence, the time-series unit is used to extract graph-level risk feature vectors in the time dimension. Lightweight regression units are used to perform nonlinear dimensionality reduction and signal mapping on graph-level risk feature vectors, and activation functions are combined to calculate and output real-time risk prediction values.

5. The low-latency interactive control method for unmanned vending machines based on edge computing according to claim 4, characterized in that, The method for generating the graph-level embedding sequence includes: The interaction intent weights in the weighted adjacency matrix are introduced as prior knowledge into the calculation process of the spatial attention unit attention to obtain the fusion attention coefficients. After obtaining the fusion attention coefficients, the spatiotemporal graph attention network performs weighted aggregation of the neighborhood information of the nodes to generate updated node state vectors, which are then aggregated into graph-level embedding vectors through global average pooling.

6. The low-latency interactive control method for unmanned vending machines based on edge computing according to claim 4, characterized in that, The interaction intent weight is a node and nodes The interaction intention weights between nodes are used to divide the weighted adjacency matrix into virtual intention edges, physical topology edges, and zero-weight edges based on node type. The virtual intent edge, if node It belongs to the set of nodes of the hand skeleton, and the nodes If it belongs to the product node set, then the interaction intent weight is the hand-object interaction intent weight; If node It belongs to the set of nodes of the hand skeleton, and the nodes If a node belongs to a risk area node set, then the interaction intent weight is the same as the interaction intent weight of the risk area. The physical topology edge, if node and nodes If all nodes belong to the hand skeleton node set and are physically connected in the skeletal topology of the hand skeleton node set, then the interaction intent weight is set to a fixed non-zero value. For all unrelated node pairs, the zero-weight edge has its interaction intent weight set to 0.

7. The low-latency interactive control method for unmanned vending machines based on edge computing according to claim 1, characterized in that, The method for automatically reducing the intensity of interactive intervention includes: Based on the intensity of the interactive intervention, it is mapped to an interactive intervention level through a preset piecewise nonlinear mapping function. If the intensity of the interactive intervention is less than the low-level threshold, it is determined to be a state of no intervention; if the intensity of the interactive intervention is not less than the low-level threshold and less than the medium-level threshold, it is determined to be soft guidance; if the intensity of the interactive intervention is not less than the medium-level threshold and less than the high-level threshold, it is determined to be a moderate warning; if the intensity of the interactive intervention is greater than or equal to the high-level threshold, it is determined to be strong intervention. The system obtains the level of interactive intervention and generates a hardware control instruction set through a control strategy selector. The hardware control instruction set includes visual control instructions, auditory control instructions, and system control instructions. If no intervention is determined, all three instructions are set to "no operation". If soft guidance is determined, only the visual control instructions are activated. If moderate warning is determined, only the auditory control instructions are activated. If strong intervention is determined, all three instructions are activated in parallel. Based on the real-time risk prediction value generated by resenting after the execution of the hardware control instruction set, combined with the interactive intervention intensity generated by recalculation, it is determined whether the interactive intervention intensity converges to a safe state. If it converges, the hardware control instruction set is automatically released.

8. A low-latency interactive control system for unmanned vending machines based on edge computing, used to implement the low-latency interactive control method for unmanned vending machines based on edge computing as described in any one of claims 1-7, characterized in that, The system includes a front-end perception module, a risk prediction module, a risk control module, and a tiered intervention module. The front-end perception module is used to collect multimodal data streams including two-dimensional color image data and three-dimensional point cloud data. Based on the hand bounding box and hand detection confidence obtained in real time from the two-dimensional color image data, it combines multi-order frequency control based on motion sensitivity to judge the hand interaction state and generate a dynamic processing frequency signal. If the dynamic processing frequency signal is greater than zero, it uses the spatial prior information provided by the hand bounding box to extract and fuse the node set of the hand, the product, the risk area, and the hand motion state set to construct a dynamic spatiotemporal map. The risk prediction module is used to acquire the dynamic spatiotemporal graph sequence within the time-series sliding window, extract graph-level risk feature vectors in the spatiotemporal dimension through a spatiotemporal attention network based on the interaction intent weights in the weighted adjacency matrix, and output real-time risk prediction values ​​based on the graph-level risk feature vectors through a lightweight regression unit. The risk control module generates the intensity of interactive intervention based on real-time risk prediction values ​​and preset risk safety thresholds by performing multiple processes, including real-time difference calculation, parallel calculation of control components, linear superposition, and saturation function constraints. The graded intervention module is used to determine the level of interaction intervention based on the intensity of the interaction intervention through a preset piecewise nonlinear mapping function, generate and execute hardware control instructions through a control strategy selector according to the level of the interaction intervention, and feed back the intensity of the interaction intervention generated by re-sensing and re-computation to determine whether convergence has occurred. If convergence reaches a safe state, the intensity of the interaction intervention is automatically released.

Images