Agent self-recovery execution method and system based on execution trajectory

Abstract

The application provides an agent self-recovery execution method and system based on execution track, and belongs to the field of safety monitoring; the problem of poor supermarket safety monitoring effect is solved; the specific method is as follows: a monitoring video is acquired, the behavior of a person in the monitoring video is analyzed, and it is judged whether the person in the video has abnormal behavior; if not, no processing is performed; if yes, a sample video is intercepted in the monitoring video; the sample video is preprocessed, multi-channel features are output, and an identification model is trained to convert the multi-channel features into atomic events; the atomic events are converted into a primary behavior chain, part of the actions in the primary behavior chain are merged, and a basic behavior chain is supplemented; abnormal behaviors in the monitoring video are detected, and different patrol routes are formulated; the application analyzes the behavior of a customer in a monitoring video through an agent, judges whether the customer behavior is abnormal, and when the agent detects a brand-new behavior chain, the agent is adaptively updated to expand the detection range of abnormal events.

Description

Technical Field

[0001] This invention relates to a method and system for intelligent agents to self-recover execution based on execution trajectories, and belongs to the field of security monitoring. Background Technology

[0002] Existing methods or systems for preventing theft in supermarkets have the following shortcomings: Traditional security relies heavily on post-incident tracing: The traditional supermarket anti-theft system is a typical "passive, post-incident" security model. It relies on real-time manual monitoring, post-incident video review, EAS radio frequency alarms at the checkout, and on-site manual catching of stolen goods. The entire prevention and control mechanism has an inherent time lag and cannot achieve early warning and in-process interception.

[0003] The fixed rule recognition mode is rigid: the abnormal behavior recognition of traditional supermarkets all adopt the judgment mechanism written by humans. The behavior samples, judgment thresholds and state jump logic are all fixed. It can only recognize a few basic theft behaviors preset by the system. It does not have the ability to learn and iterate and update on its own, and cannot adapt to the continuously iterative and upgraded theft methods.

[0004] Traditional fixed patrol mode has low manpower utilization: Traditional supermarket patrols rely entirely on fixed routes, fixed time periods, and fixed areas for manual patrols. They do not dynamically adjust patrol strategies based on regional risk levels, historical theft data, and real-time abnormal risks. This can easily lead to insufficient patrols in high-risk areas and repeated backtracking in low-risk areas. It cannot simultaneously meet the needs of emergency response and full coverage patrols of the merchandise area, thus preventing limited security personnel from maximizing their prevention and control value. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the present invention aims to provide a method and system for intelligent agent self-recovery execution based on execution trajectory, in order to solve the problem of poor safety monitoring performance in supermarkets.

[0006] To achieve the above objectives, the present invention provides a self-recovery execution method for intelligent agents based on execution trajectories, comprising: Define the basic behavior chain and build a state machine; acquire surveillance video, analyze the behavior of people in the surveillance video, and determine whether there is any abnormal behavior in the video; if not, do not process it; if so, extract sample video from the surveillance video. The sample video is preprocessed to output multi-channel features; an image library of different human actions is set up to train a recognition model for recognizing human actions; the multi-channel features are converted into atomic events through the recognition model; the atomic events are converted into primary behavior chains, and some actions in the primary behavior chains are merged to supplement the basic behavior chains. Obtain historical anomaly data and calculate the risk coefficient of each area to be patrolled; detect whether there is any abnormal behavior in the surveillance video; if not, formulate a patrol route according to the risk coefficient; if so, immediately alarm and record the abnormal location, and patrol the area where the abnormal location is located.

[0007] Furthermore, the steps for judging abnormal behavior are as follows: Define the basic behavior chain, and based on the basic behavior chain, define the state and state transition conditions; Match motion clips of each person in the surveillance video, identify each person's actions, and output multi-dimensional real-time features of each person. Based on each person's traceable ID, create an independent state machine instance for each person, and match each person's real-time state while shopping in the supermarket according to the state transition determination rules. Define a normal shopping state transition path as the standard state transition path, and detect the state changes of each person during shopping; Scenario 1: If a person has fully experienced the state changes corresponding to the standard state transition path, and the person's state has never been in a warning state or an undefined state, then the person is normal. Scenario 2: If a person has experienced the state change corresponding to the standard state transition path, but the person's state has changed to the alert state, then in the state machine instance corresponding to the person, it is determined whether the person's last state change matches the alert cancellation path. If a match is found, then the person is normal; If there is no match, the person is suspected of being suspicious. Scenario 3: If the change in a person's status does not meet the requirements of Scenario 1 and Scenario 2, then it is determined that the person has exhibited abnormal behavior, and an alarm should be raised immediately. In real time, video segments corresponding to scenarios two and three are extracted from the surveillance video and used as sample videos to fit a new behavior chain.

[0008] Furthermore, the fitting steps for the new behavioral chain are as follows: Extract the bounding box of each person's body from the sample video; based on the bounding box, crop each video frame in the sample video pixel by pixel to obtain the ROI image of each person; Identify skeletal key points in the pixel images of each person's body, and for each person... The image undergoes neutral alignment, size normalization, and temporal unification processing. Will The horizontal pixel coordinates, vertical pixel coordinates, and visibility of the image are concatenated as three independent channels to obtain a standardized sample image tensor. Train a recognition model to identify human actions. Input sample image tensors into the recognition model. Based on the position of each person in the sample video and the actions recognized by the model, output the atomic events of each person in the sample video. Arrange the actions of each person in the atomic event in chronological order to obtain the primary behavior chain.

[0009] Furthermore, the fitting steps for the new behavioral chain also include: Get the number of actions in the template chain ; In the sample video, select behavior chain b; Get the number of actions in behavior chain b Set the threshold for normalized similarity. ; Transform all actions in behavior chain b and template chain into semantic vectors; Based on the number of actions in behavior chain b and template chain, set up a baseline chain and a control chain; The number of identical semantic vectors in the baseline chain and the control chain. Calculate the normalized similarity between behavior chain b and template chain. ,judge Is it greater than or equal to? ; If it is greater than or equal to, it means that the customer's primary behavior chain and basic behavior chain are highly similar. The existing basic behavior chain will be used directly for judgment, and no action will be taken. If the similarity is less than 0, it indicates that the customer's primary behavior chain and basic behavior chain have low similarity. Based on all customer actions in behavior chain b, the encoder stage of the recognition model extracts the corresponding values ​​for each customer action. The diagram shows the merging of actions in behavior chain b.

[0010] Furthermore, the steps for merging actions in behavior chain b are as follows: Select two time-continuous frames in behavior chain b. Diagram, denoted as diagram and picture ; Get Image Previous frame ; Constructing a graph The three-dimensional features include: spatial features, action features, and interaction features. Get Image The first two and first four frames of images are used to obtain the image. and picture ; Get Image and picture The three-dimensional features, in the figure To the map Then to the picture Based on the time span, calculate the graph Curvature corresponding to spatial features motion characteristic curvature curvature of interactive features ; Calculation graph The curvature corresponding to spatial features, action features, and interaction features is obtained. , and ; Set the curvature determination error, and judge sequentially. and , and as well as and Are all the differences within the curvature determination error? If so, then retain the graph in behavior chain b. Corresponding atomic events, delete graph The corresponding atomic events; if none or only some are present, then the graph is preserved in behavior chain b. and picture The corresponding atomic events.

[0011] Furthermore, the steps for developing patrol routes are as follows: Calculate the risk coefficient for each area to be patrolled based on historical anomaly data; The risk coefficients of each patrol area are arranged in ascending order from smallest to largest. Using the 15th and 85th percentiles as quantiles, the threshold for determining the risk coefficient of the commodity area is extracted, thus obtaining the threshold. With threshold ; Risk coefficient greater than or equal to the threshold The areas to be patrolled are designated as high-risk areas; areas with a risk coefficient less than or equal to the threshold are designated as high-risk areas. The areas to be patrolled are designated as low-risk areas; the remaining areas to be patrolled are designated as medium-risk areas. Detect any unusual behavior in surveillance videos; If not, then patrol the commodity areas corresponding to high-risk, medium-risk, and low-risk areas in sequence according to the size of the risk coefficient; If so, record the abnormal location, determine whether the abnormal location has changed, and formulate different branch patrol plans; Branch 1: If the abnormal location remains unchanged, use reverse update to determine the patrol route; Branch 2: Abnormal location changes, increase the number of patrol personnel, predict abnormal location changes, and formulate patrol routes.

[0012] Furthermore, the steps for branch one are as follows: Obtain the location of patrol personnel and any unusual locations, and record them as locations. With position ; Obtain the location of each area to be patrolled, and record it as the location. ~location Establish a data stack and store the locations of all areas to be patrolled into the stack; Calculate the position sequentially Arrive at the location ~location The route length is obtained. ~ ;extract ~ minimum value ,Will The corresponding area to be patrolled is designated as the first patrol zone; the position of the first patrol zone is removed from the data stack; The areas outside the first patrol zone are designated as secondary patrol zones; the route length from the first patrol zone to each secondary patrol zone is calculated sequentially. ~ ;extract ~ minimum value ,Will The corresponding secondary patrol area is moved out of the data stack as the second patrol area; Repeat the reverse update steps for finding the first and second patrol zones, and search for the third to fourth patrol zones in the remaining unpatched zones. Patrol the area until the data stack is empty; Record the pop order of each area to be patrolled in the data stack, arrange all areas to be patrolled in the pop order to obtain the primary patrol route; reverse the primary patrol route to obtain the patrol route.

[0013] Furthermore, the processing steps for branch two are as follows: Get current time Based on the current time of the anomaly's location, predict its positional changes until it is predicted to reach the patrol area, thus obtaining the first target area; record the first reference time when the anomaly moves to the first target area. ; Identify the patrol area corresponding to the primary target area as the key area; acquire anomaly locations. , distance position The nearest product area is used as the initial target area, and the patrol area corresponding to the initial target area is used as the initial area. Determine whether the initial area and the key area are the same area, and execute different sub-branch patrol strategies accordingly; Sub-branch 21: If the initial region and the key region are the same region, then the region outside the key region is treated as a secondary region, based on the current time. Compared with the first reference time Adjust patrol routes in key areas based on time differences; Sub-branch 211: The path from the initial target area to the first target area is designated as the primary patrol route, and patrol personnel will patrol at the first reference time. Upon reaching the initial target area, patrol the key areas along the designated patrol route; Sub-branch 212: Calculate the time cost of each path in the key patrol area and set the patrol route based on the time cost; Sub-branch 22: If the initial region and the key region are not the same region, then the path from the initial region to the key region is taken as the quasi-path; based on the current time... Compared with the first reference time Adjust patrol routes in key areas based on time differences.

[0014] Furthermore, the processing steps for sub-branch 212 are as follows: Sub-branch 212: Obtain the location of patrol personnel in key areas and count the number of merchandise areas in key areas. The initial target area is designated as region C, and the first target area is designated as region D. Obtain the movement speed of the patrol personnel and calculate... and time difference ; Count the number of paths in key areas Based on the patrol personnel's movement speed, calculate the number of patrol personnel passing through the key areas from the 1st to the 2nd. The travel time of each path is obtained. ~ ; Will ~ Arrange in ascending order from smallest to largest to obtain ~ ; Compare and The size, if ≥ Then stop iterating, and Any product area along the corresponding path will serve as the starting point for patrols, and the patrol speed will be increased. like ≥ Then calculate ~ The former The sum of terms makes ~ The former The sum of the terms is greater than or equal to the sum of the terms for the first time. ; exist ~ forward Among the paths corresponding to the items, completely unreachable paths are removed, resulting in The marked path; by introducing supplementary paths into the marked path, the first patrol route of the patrol personnel is obtained; Determine if there is a common area between the first and second target paths; If present, do not process. If not, the shortest path from the first marker path to the second marker path is introduced as a parameter path; the travel time of patrol personnel along the parameter path is calculated. ; judge Is it less than or equal to the first? Travel time for individual routes; If so, then introduce the parameter path and discard the first one. Individual target path; If not, then discard the second target path; Repeat the steps of introducing the parameter path until the first patrol route is obtained.

[0015] The intelligent agent self-recovery execution system based on execution trajectory includes: Video acquisition module: used to acquire surveillance video inside the supermarket; Behavior Analysis Module: Used to define basic behavior chains, analyze the behavior of people in surveillance videos, and obtain multi-dimensional real-time characteristics of each person; based on the multi-dimensional real-time characteristics of each person, determine whether there is any abnormal behavior in the video; if not, skip the behavior chain update module; if so, extract video clips of abnormal behavior from the surveillance video as sample videos. Behavior chain update module: used to preprocess sample videos and output multi-channel features; train a recognition model to recognize human actions, and convert the multi-channel features of people into atomic events through the recognition model; convert atomic events into primary behavior chains, merge some actions in the primary behavior chains, and supplement the basic behavior chain; Patrol planning module: Used to acquire historical anomaly data, calculate the risk coefficient of each area to be patrolled; detect whether there is abnormal behavior in the surveillance video; if not, patrol the area to be patrolled according to the risk coefficient; if so, immediately alarm and record the abnormal location in real time, and patrol the area where the abnormal location is located.

[0016] Compared with the prior art, the beneficial effects of the present invention are: Preventing Theft: This invention breaks down a complete theft into a standardized temporal state chain using a behavior chain and a finite state machine. The system extracts multi-dimensional temporal features based on key points of the human skeleton, customer movement trajectories, customer dwell time in each product area, and customer-product interaction. Subsequently, the continuity of customer behavior is verified segment by segment according to the state transition rules of the state machine. As soon as an abnormal behavior node is detected or a behavior transitions to an abnormal behavior, an alarm is immediately triggered, enabling real-time intervention and reducing supermarket losses at the source.

[0017] Adaptive Behavior Chain Construction: The adaptive behavior chain construction mechanism of this invention can continuously collect surveillance videos in supermarkets, automatically summarize newly added abnormal behavior time-series segments, and automatically expand the theft behavior chain sample library without the need for manual code to update rules.

[0018] Risk-weighted optimized patrol routes: This invention calculates the real-time risk coefficient of each commodity area based on historical anomaly data, and adopts a risk-distance fusion clustering and segmentation algorithm to dynamically divide each patrolman's exclusive non-conflict area, with the goal of minimizing the time spent by the last person to finish and evenly distributing the task load. Attached Figure Description

[0019] Other features, objects, and advantages of the invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a schematic diagram of the method steps of the present invention; Figure 2 This is a schematic diagram illustrating the functions of the modules in this invention; Figure 3 This is a schematic diagram of the behavior chain update of the present invention. Detailed Implementation

[0020] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0021] Example 1 Please see Figure 1 The agent self-recovery execution method based on execution trajectory includes: Step S1: Define the basic behavior chain and construct a finite state machine; acquire real-time surveillance video of the supermarket, analyze the behavior of each customer in the surveillance video, and obtain multi-dimensional real-time features of each customer; based on the multi-dimensional real-time features of each customer, assign an independent state machine instance to each customer and determine whether there is any abnormal behavior in the supermarket; if not, skip step S2 and proceed to step S3; if so, extract the video clip with theft behavior from the surveillance video as a sample video. It should be noted that the abnormal behavior in step S1 represents theft. Type A intelligent agent: used to acquire all surveillance videos in the supermarket and analyze customer behavior in the videos to monitor for theft prevention of goods; Type B intelligent agents: used to calculate the risk coefficient of each product area in the supermarket and to formulate patrol routes within the supermarket; Define two basic behavioral chains: the normal shopping behavior chain and the theft behavior chain; Define the normal shopping behavior chain: enter the supermarket → browse and select products in the product area → take the products → go to the cashier to pay → walk to the supermarket entrance; Define the chain of theft: entering the supermarket → lingering in the merchandise area for an extended period of time → taking items → hiding items → walking directly to the supermarket entrance without going to the checkout counter; Based on the basic behavior chain, define 6 states: state Enter the supermarket; state : In the shopping area; state Take the goods; state Concealing goods; state Go to the cashier to pay; state : Walk towards the supermarket entrance; Based on the existing state, define 9 paths: path Customers enter the supermarket and linger in the merchandise area; Path 1; path Path 2: Customers linger in a product area for an extended period of time before taking items from it; If a customer stays in a product area for more than 3 minutes, it is determined that the customer has been lingering in the product area for an extended period of time. path : After receiving the goods, the customer hides them; Path 3; path The customer takes the concealed goods to the cashier to pay; Path 4; path After checking out, the customer walks towards the supermarket entrance; Path 5; path Customers enter the supermarket and head straight to the merchandise area to pick up their items; Path 6; path After receiving the goods, the customer goes to the cashier to pay; Path 7; path Customers hide the goods and walk directly to the supermarket entrance without paying; Path 8; path Customers take the goods and walk directly to the supermarket entrance without paying; Path 9; path The customer takes out the hidden item; Path 10; Define the rules for determining state transitions: Any customer's state as long as the path has been executed And it has not reached that state. If so, the customer is considered to be making a normal purchase. Any customer's state can be changed to state Therefore, it was determined that the customer was suspected of theft; Any customer's state can be changed to state or state And the path is not executed. If the customer is found guilty of theft, call the police immediately. Define the state transition conditions for path 1 as follows: the customer stays in a certain product area for more than 12 seconds, the customer's movement speed in the product area is less than 3 meters / second, during the time the customer stays in the product area, the average distance between the customer and the supermarket staff is greater than or equal to 1 meter per second, and the customer does not make any picking up actions within 10 consecutive seconds during the time the customer stays in the product area. Define the state transition condition for path 2: If a customer stays in a certain product area for more than 3 minutes and makes a purchase, record the number of product categories the customer takes. and total amount of goods ; Define the state transition conditions for path 3: After the customer receives the goods, the distance between the customer and the supermarket staff is greater than or equal to 1 meter, and the customer engages in actions of obscuring or hiding; record the remaining quantity of goods in the customer's shopping cart or basket. ,like Greater than Then determine the customer's state. Become a state ; like equal Then record the number of remaining product categories in the customer's shopping cart or basket. , judgment and Are they equal? ​​If they are equal, then the customer remains in the same state. If they are not equal, then the customer is determined to be in a certain state. Become a state ; Define the state transition condition for path 4: The customer has performed actions of obscuring and hiding items before checking out at the cashier, and the customer has not taken any items out of the hidden area. Record the actual number of item categories at the time of checkout. and the actual quantity of goods ; judge and Are they equal? ​​If and If they are equal, then judge. and If they are equal, change path 4 to path 10; if they are not equal, the customer is suspected of theft. like and If they are not equal, the customer remains in their current state. ; Define the state transition condition for path 5: After the customer pays at the checkout counter, they walk towards the supermarket entrance; Define the state transition conditions for path 6: After entering the supermarket, the customer keeps walking around, and the customer stays in each product area for less than 8 seconds; the customer only takes items during the walking process. Define the state transition condition for path 7: The customer only takes action and does not cover or hide anything while the customer is going to the cashier. Define the state transition condition for path 8: The customer has performed actions of obscuring and hiding, and the customer has not gone to the cashier to pay, but has gone directly to the supermarket entrance; Define the state transition condition for path 9: The customer only takes something and does not go to the cashier to pay, but walks directly to the supermarket entrance; Define the state transition condition for path 10: The customer has performed an obscuring and hiding action, and the customer takes out the product from the hidden area; The system acquires real-time surveillance video from the supermarket, uses the YOLOv5s model to identify and extract the visual features of each customer, and then uses a multi-object tracking algorithm to assign a unique traceable ID to each customer in the supermarket. The visual feature vectors are preferably selected based on the customer's body shape features, clothing texture features, and walking gait features. Users or relevant technical personnel can set other visual features according to actual needs. The multi-object tracking algorithm is preferably the ByteTrack algorithm. Based on visual features extracted using the YOLOv5s model, the DeepSORT algorithm is used to automatically match motion segments of each customer in different surveillance videos, and then stitch these motion segments together to obtain each customer's movement route within the supermarket. Based on the movement routes obtained by the DeepSORT algorithm, the YOLOv8s model is used to locate the product positions in each frame of the surveillance video in real time, and the ByteTrack tracking algorithm is used again to track the product positions across frames. Finally, the YOLOv8n-pose lightweight model is used to identify the actions of each customer in the surveillance video, outputting multi-dimensional real-time features for each customer. These multi-dimensional real-time features specifically include: motion features, scene features, distance features, product features, and behavioral features. Motion characteristics represent: the real-time location of customers within the supermarket and their movement speed within the supermarket; Scene feature representation: the time customers spend in different product areas of the supermarket; Distance feature representation: the distance between a customer and a supermarket staff member or the distance between a customer and the checkout counter during a supermarket shopping process; Product characteristics: the type and quantity of products taken by the customer; Behavioral characteristics include: customers' actions of taking things, covering things, and hiding things; Identify the take action: The model identifies the positions of a customer's shoulders, elbows, wrists, torso, and hands in surveillance video. Define the rules for determining the take action: rule The angle formed by the customer's shoulder, elbow, and wrist is greater than 120 degrees; rule The customer's wrist position moves from the torso area to the shelf area of ​​the merchandise area; rule The customer's wrist is withdrawn from the shelf area, and there are products in the area of ​​the customer's palm; In the surveillance video, when a customer simultaneously meets the rule for 6 consecutive frames... and This means that the customer has performed a take action; Identify occlusion actions: The model identifies the location of a customer's wrist, torso, face, palm, and merchandise in surveillance video. Define the rules for judging occlusion actions: rule The model was unable to recognize the customer's facial area. rule The customer's palm area contains merchandise, and the customer's palm area moves to the customer's torso area; rule The customer's wrist and merchandise could not be detected in the customer's torso area; In the surveillance video, when a customer simultaneously meets the rule for 5 consecutive frames... to This means that the customer is determined to have made an obstructive action. Identify concealment actions: Designate a hiding area: The customer's torso and non-supermarket-provided storage items are designated as hiding areas; non-supermarket-provided storage items refer to items that customers carry with them, such as backpacks, handbags, and school bags, which can be used to store goods. Define the rules for judging the act of hiding: rule The customer simultaneously performs the actions of taking something and covering it up. rule The customer has merchandise in their palm area, and then moves their palm holding the merchandise to a hidden area; rule The customer's palm lingers in the hidden area for more than 0.8 seconds, or 20 frames. In the surveillance video, when a customer simultaneously meets the rule for 20 consecutive frames... to This means that the customer was determined to have engaged in concealment. A type of intelligent agent creates an independent state machine instance for each customer based on each customer's traceable ID. This state machine instance is used to record the state changes of each customer in each frame during the actual shopping process; the initial state of the state machine instance is state. When a customer walks towards the supermarket entrance, if the customer is not suspected of theft or is identified as a thief, then the customer's state machine instance is destroyed 8 seconds after the customer walks towards the supermarket entrance. Class A agents match the real-time state of each customer while shopping in the supermarket based on the state transition determination rules and the real-time features output by the YOLOv8n-pose lightweight model. Define the state transition path for normal shopping: State →Status →Status →Status , or state →Status →Status →Status →Status ; The normal shopping state transition path is used as the standard state transition path to detect the state changes of each customer during shopping; Scenario 1: If the customer has fully experienced the state changes corresponding to the standard state transition path, and the customer's state has never changed before checking out at the cashier. If a customer's status has never been undefined, then the customer is considered to be making a normal purchase. Here, an undefined status indicates that the customer does not meet the above criteria. To state Defined state; Scenario 2: If the customer has experienced the state change corresponding to the standard state transition path, but the customer's state changed before checking out at the cashier. Then, in the customer's corresponding state machine instance, it is determined whether the customer's last state change matches path 10; the aforementioned alert state represents the state. The above-mentioned alert clearance path represents path 10; If a match is found, the customer is making a normal purchase and falls under scenario one; If the match is not found, the customer is suspected of theft. Scenario 3: If the customer's status changes do not meet the requirements of Scenario 1 and Scenario 2, it is determined that the customer has committed theft, and the police should be notified immediately. Scenario 4: If the customer has experienced the state change corresponding to the standard state transition path, but the customer's state changed to an undefined state before checking out at the cashier, then it is determined that the customer is in an undefined state. Whether the actual product categories and quantities taken are consistent with those scanned by the barcode scanner at checkout; If they match, then the customer is suspected of theft, falling under scenario two; If there is no discrepancy, the customer has committed theft, and the police should be notified immediately, which falls under scenario three; Extract video clips corresponding to scenarios two and three from the supermarket's surveillance video in real time as sample videos; analyze the theft behavior in the sample videos and fit a new theft behavior chain.

[0022] Step S2: Preprocess the sample video to output multi-channel features; set up an image library of different human actions and train a recognition model to recognize human actions; convert the multi-channel features of customers into atomic events of customers through the recognition model; convert the atomic events into primary behavior chains, merge some actions in the primary behavior chains to obtain a supplementary chain of the basic behavior chain. Please see Figure 3 From the sample videos, select customers, backpacks, shopping carts or baskets, goods, shelves, cash registers, supermarket staff, and supermarket entrances / exits as observation targets; In the sample video, each customer entering the supermarket is assigned a unique [personal identifier / feature]. It is used to track each customer across shots in the sample video to ensure the continuity of the time sequence; Based on the layout information of each product area in the supermarket, a unique spatial label is assigned to each product area in the sample video to represent the pixel position of each product area in each sample video. The YOLO object detection model was used to process the sample video to obtain the bounding box of each customer's body. Based on the bounding box of each customer's body, each video frame in the sample video was cropped pixel by pixel to obtain the local pixel image of each customer's body, that is, the human body ROI image. Based on the MediaPipe Pose technology, the skeletal key points of each customer were identified in the human body pixel image of each customer. The main identification was of the skeletal key points of each customer's head, shoulders, torso, and arms. These four types of skeletal key points are related to theft actions. The main skeletal key points identified include: Key points of the head (5 in total): Tip of the nose, inner corner of the left eye, outer corner of the left eye, inner corner of the right eye, outer corner of the right eye; Function: Used to calculate the customer's head tilt angle and orientation; Key points of the upper body, shoulders, neck, and torso (12 in total): Left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, heel of left hand, heel of right hand, left hip (left side of waist), right hip (right side of waist), midpoint of spine, root of neck; Function: Used to calculate the customer's lateral twisting angle and arm flexion / extension angle; Key points for the left hand (5 in total): tip of the left thumb, tip of the left index finger, tip of the left middle finger, tip of the left ring finger, tip of the left little finger; Function: Used to determine whether a customer's hand area is obstructed, and whether they are taking or hiding products; Key points for the right hand (5 in total): tip of the right thumb, tip of the right index finger, tip of the right middle finger, tip of the right ring finger, tip of the right little finger; Function: Used to determine whether a customer's hand area is obstructed, and whether they are taking or hiding products; Neutral alignment: This refers to the local pixel image of each customer's body as... Images, in each customer's In the image, the midpoint of the spine corresponding to each customer is selected as the alignment reference point; in each customer's... In the image, the relative coordinates of the remaining 26 skeletal key points of each customer with respect to their alignment reference point are calculated to eliminate the interference caused by the height of each customer and achieve the centered alignment of the 27 skeletal key points of each customer. Size normalization: for each customer In the image, extract the pixel coordinates of the left and right shoulders (which are not centered and aligned), calculate the pixel distance between the left and right shoulders for each customer, and use this distance as the horizontal pixel width for each customer; then set a globally universal standard horizontal pixel width. ; standard width Using the horizontal pixel width of each customer as the numerator and the pixel scaling ratio of each customer as the denominator, the pixel scaling ratio of each customer is calculated. Using the pixel scaling ratio of each customer, the pixel coordinates of the 27 centrally aligned skeletal key points corresponding to each customer are scaled proportionally to achieve size normalization of all customers. Timing unification: Using a linear interpolation algorithm to match customers The frame rate of the image has been adjusted to 64. Acquiring customers Number of frames in an image Calculate customers Image temporal mapping ratio : ; Calculate the customer in the s-th second The floating-point temporal position of the image in the i-th frame : ; Where i represents the frame number, and the value of i ranges from 1 to 64; s represents the time. extract integer part and decimal part For customers The image in the i-th frame at second s Linear interpolation of the image: In customers Image at the s second In the frame image, select the first one. Frame image as the difference of the i-th frame The previous frame of the image is denoted as image . ; will the first Frame image as the difference of the i-th frame The next frame of the image is denoted as image. ; Linear interpolation of the left shoulder pixels: Assuming that after linear interpolation, the i-th frame at second s... The pixel coordinates of the left shoulder in the image are ; Let the image The left shoulder pixel coordinates are ,image The left shoulder pixel coordinates are , The formula for calculation is: ; Repeat the linear interpolation process on the left shoulder pixels for the customer. Linear interpolation is performed on the pixel coordinates of other pixels in the image, and the interpolated pixel coordinates are then centered and aligned. Customers The horizontal pixel coordinates, vertical pixel coordinates, and visibility of the image are concatenated as three independent channels to obtain a standardized sample image tensor; here, the customer... The image is a customer image that has been center-aligned, size-normalized, and time-unified. image; Training a recognition model to identify human movements: Images with various action name labels are obtained as training samples; each image in the training samples is processed in the same way as the sample video, including center alignment, size normalization and temporal unification; for each processed training sample image, all images in the training sample that are different from that training sample image are used as negative samples of that training sample image. For each processed training sample image, different enhancement strategies are used to generate enhanced images of the same training sample image, which are used as positive samples of that training sample image; the number of enhanced images corresponding to the same training sample image is at least two, and the optional enhancement strategies include: temporal enhancement, spatial enhancement and noise enhancement; The encoder for training the model: A spatiotemporal graph convolutional network is used as the backbone network to train an encoder that recognizes changes in human skeletal movements; the preferred spatiotemporal graph convolutional network is ST-GCN. Based on the actual connection relationships of the skeletal keypoints, an adjacency matrix for the skeletal keypoint connections is constructed to replace the original 17 / 33 node structure of the ST-GCN network. The size of the adjacency matrix is ​​27×27, and the rows and columns in the adjacency matrix represent the keypoints corresponding to the head, shoulders, torso, and arms, respectively. In the adjacency matrix, if any two skeletal keypoints are connected, the corresponding matrix element of these two skeletal keypoints is one; otherwise, the corresponding matrix element of these two skeletal keypoints is zero. The ST-GCN network extracts the motion patterns of key points of bones in consecutive frames for the same action through spatial graph convolution S-GCN and temporal convolution TCN, resulting in a standardized 3×64×27 bone tensor. The 3×64×27 bone tensor is then output as a 512-dimensional bone feature vector through global average pooling. A projection head consisting of two fully connected layers and a ReLU activation function is connected after the backbone network to map the 512-dimensional skeletal feature vectors to a 128-dimensional low-dimensional space. The InfoNCE function is used as the loss function and AdamW is used as the optimizer to compare and learn positive and negative samples of the same action until the cosine similarity between two feature vectors generated by positive samples in the same action is 1 or greater than 0.85, and the cosine similarity between any feature vectors generated by any negative sample of the same action is 0 or less than 0.15, thus obtaining an encoder for recognizing changes in human skeletal movements. Decoder for training model: The trained encoder that recognizes changes in human skeletal movements processes the images in the training samples again to obtain the feature vectors of each image in the training samples. Based on the various action labels in the training samples, a primary summary is performed on all feature vectors in the training samples to obtain primary feature vector clusters for each type of action. In the primary feature vector clusters corresponding to each type of action, a clustering algorithm is used to extract the center vector of each cluster. The preferred clustering algorithm is the HDBSCAN density clustering algorithm. Based on the center vector of each cluster, feature vectors with a cosine similarity of less than 0.5 to the center vector are removed from each cluster to obtain standard feature vector clusters for various actions; each standard feature vector cluster corresponds to a specific action name label. For each type of action appearing in the training samples, all feature vectors in the standard feature vector cluster corresponding to each action name label are taken as positive samples for that action, and other standard vector clusters that are different from that action are taken as negative samples. The SkeletonCLIP text encoder is used as the basic decoder for the encoder that recognizes changes in human skeletal movements, and the InfoNCE function is used as the loss function to train the SkeletonCLIP text encoder. During the training of the decoder, a BERT module supporting Chinese is introduced, and the descriptive text of the action name tags corresponding to each standard feature vector cluster is used as the training object. During the specific training, the action name tags are first preprocessed by word segmentation, filtering of stop words, and truncation and padding, and the action name tags corresponding to each standard feature vector cluster are padded to 64 characters. The semantic vectors of each type of action name tag after preprocessing are extracted, and global average pooling and linear layer mapping are used to align the dimensions of the action name tag semantic vectors with the encoder output feature vectors. During training, a new action classification head is added, which maps the feature vector output by the encoder to the probability distribution of all action name categories. Then, the semantic vector of the action name label and the feature vector output by the encoder are trained synchronously. The total loss is composed of the weighted sum of the InfoNCE loss and the cross-entropy loss of the action classification, until the maximum number of iterations preset by the user is reached. After training, the feature vector output by the encoder is fed into the classifier head, which outputs the action category with the highest probability and maps it to the corresponding action name. The standardized sample image tensor, converted from the sample video, is input into the human motion recognition model. Based on the position of each customer in the sample video and the customer's actions recognized by the model, the model outputs atomic events for each customer in the supermarket. These atomic events include: time, and the customer's uniqueness. Customer location, customer actions, and the items the customer takes; According to the customer's unique Arrange the customer actions in each customer's atomic events in chronological order to obtain the customer's primary behavior chain; Use the theft chain as a template chain; Get the number of actions in the template chain, denoted as ; In the sample video, a customer is randomly selected as Customer A; Let customer A's initial behavioral chain be denoted as behavioral chain b; Get the number of actions in behavior chain b Set the threshold for normalized similarity. ; The value range is 0.5 to 1. 0.7 is preferred; Traverse all actions in behavior chain b and template chain, and calculate the longest common sequence chain of behavior chain b and template chain: Using the BERT model, the behavior chain b and all actions in the template chain are transformed into semantic vectors; Compare the number of actions in behavior chain b with the number of actions in the template chain. Use the chain with fewer actions as the baseline chain and the chain with fewer actions as the control chain. Count the number of identical semantic vectors in the baseline chain and the control chain, denoted as . ; Calculate the normalized similarity between behavior chain b and template chain. : ; judge Is it greater than or equal to? If it is greater than or equal to, it means that the customer's primary behavior chain and basic behavior chain are highly similar. The existing basic behavior chain will be used directly for judgment, and no action will be taken. If the similarity is less than 0, it indicates that the customer's primary behavior chain and basic behavior chain have low similarity. Based on all customer actions in behavior chain b, the encoder stage of the recognition model extracts the corresponding values ​​for each customer action. The diagram shows the merging of actions in behavior chain b. Select two time-continuous frames in behavior chain b. Diagram, denoted as diagram and picture ; Get Image Previous frame Diagram, denoted as diagram ; Constructing a graph The three-dimensional features, figure The three-dimensional features include: spatial features, action features, and interaction features; Computational spatial features: Obtaining the graph The center coordinates of the customer Get the image The center coordinates of the customer ; where the center coordinates represent The pixel coordinates of the midpoint of the customer's spine in the image; Calculate coordinates relative coordinates The pixel movement distance, as the image Spatial characteristics; Get Image The coordinates of 27 key skeletal points of Customer A were obtained to create an image. The coordinates of 27 skeletal key points of Customer A are calculated from the figure. Transform into a diagram The average pixel movement distance, as the image The characteristics of the movement; Get Image The coordinates of the left palm heel of customer A are obtained from the image. The center coordinates of various goods are calculated, and the average pixel distance from the coordinates of the base of customer A's left hand to the center coordinates of various goods is used as the graph. Interactive features; Repeatedly retrieve images Steps for obtaining 3D features, including image acquisition Three-dimensional features; Get Image The first two and first four frames are denoted as images respectively. and picture ; The diagram The three-dimensional features, denoted as ;in, The diagrams are shown in sequence. Spatial features, action features, and interaction features; Repeatedly retrieve images Steps for obtaining 3D features, including image acquisition The picture The three-dimensional features are denoted as follows: and ; The diagrams are shown in sequence. Spatial features, action features, and interaction features; The diagrams are shown in sequence. Spatial features, action features, and interaction features; With diagram To the map Then to the picture Based on the time span, calculate the graph Curvature corresponding to spatial features : ; Calculation graph Curvature corresponding to action features : ; Calculation graph Curvature corresponding to interactive features : ; Repeated calculation to The steps are to calculate the graph sequentially. The curvature corresponding to spatial features, action features, and interaction features is obtained. , and ; Set the curvature determination error, and judge sequentially. and , and as well as and Are all the differences within the curvature determination error? If so, then retain the graph in behavior chain b. Corresponding atomic events, delete graph The corresponding atomic events; if none or only some are present, then the graph is preserved in behavior chain b. and picture The corresponding atomic events; Repeat the steps of processing the initial behavior chain of customer A, merge the actions in other customer behavior chains in the sample video, and after merging, use the Hopcroft algorithm to perform a second merging to obtain a new theft behavior chain; use the new theft behavior chain as a supplementary chain to the basic behavior chain.

[0023] Step S3: Obtain historical anomaly data from the supermarket and calculate the risk coefficient of each product area in the supermarket; detect whether there is theft in the surveillance video in real time; if not, patrol the product area according to the risk coefficient; if so, immediately alarm and record the location of the thief in real time, and patrol the area where the thief is located; the above abnormal location indicates: the location of the thief. Obtain historical theft data from the supermarket. Historical theft data includes: the number of thefts, the amount stolen, and the stolen goods; the aforementioned historical anomaly data also represents historical theft data. Based on the supermarket's historical theft data, the risk coefficient of each merchandise area in the supermarket is calculated; the above-mentioned areas to be patrolled represent merchandise areas. Count the number of product sections in the supermarket ; Count the number of times each type of product was stolen from historical theft data, and calculate the theft coefficient of each type of product: Theft coefficient of a product = Number of times a product was stolen ÷ Time the product was put on the shelf; In historical theft data, the total value of stolen goods for each type is tallied, and the average value of stolen goods for each type is calculated based on the number of times each type of goods is stolen. The theft coefficient and average theft amount for each product are standardized. The sum of the standardized theft coefficient and standardized average theft amount for each product is processed using the Sigmoid function to obtain the product risk coefficient for each product. Based on the actual types of goods in each commodity zone, the risk coefficients of each commodity in the same commodity zone are added together to obtain the risk coefficient of each commodity zone. Arrange the risk coefficients of each commodity zone in ascending order, and extract the judgment threshold for the risk coefficient of each commodity zone using the 15% and 85% quantiles. With threshold ; Risk coefficient greater than or equal to the threshold The commodity area is designated as a high-risk area; areas with a risk coefficient less than or equal to the threshold are also considered high-risk areas. The designated product areas are low-risk areas; the remaining product areas are medium-risk areas. Repeat step S1 to detect theft in the surveillance video in real time; If not, then patrol the commodity areas corresponding to the high-risk area, medium-risk area and low-risk area in descending order of risk coefficient according to the size of the risk coefficient; If so, the location of the thief is recorded in real time, it is determined whether the location of the thief has changed, different branching plans are formulated, and the area traversed by the thief is patrolled. Branch 1: The thief's location remains unchanged; use reverse update to determine the patrol route. Obtain the locations of patrol personnel and thieves, and record them as "locations" respectively. With position ; Obtain the location of each product section in the supermarket, and record it as the location. ~location ; Create a data stack and store the locations of all product areas into the stack; Calculate the position sequentially Arrive at the location ~location The route length is obtained. ~ ;extract ~ minimum value ,Will The corresponding product area is designated as the first product area; the position of the first product area is moved out of the data stack; the aforementioned first patrol area also represents the first product area; The commodity zones other than the first commodity zone are designated as second-level commodity zones; the number of second-level commodity zones is as follows: ; Calculate the route length from the first commodity zone to each second-level commodity zone in sequence, and obtain ~ ;extract ~ minimum value ,Will The corresponding product area, as the second product area, is moved out of the data stack; the aforementioned second patrol area also represents the second product area. Repeat the reverse update steps for the first and second product areas, and then sequentially search for the third to fourth product areas in the remaining product areas. The product area continues until the data stack is empty; the third to the fourth sections mentioned above... The patrol area also indicates the third to the fourth. Merchandise area; Record the pop order of each product area in the data stack, arrange all product areas in the pop order to obtain the initial patrol route; reverse the initial patrol route to obtain the supermarket's patrol route. Branch 2: The location of thieves changes; increase the number of patrol personnel, predict changes in the location of thieves, and formulate patrol routes. Get the number of patrol personnel Divide all products into There are 1 patrol area, and each patrol area is the responsibility of one patrol officer; Based on the real-time location changes of the thief in the surveillance video, Kalman filtering is used to predict the thief's moving location. Obtain the time period when the thief entered the supermarket. ~ , ~ The unit is seconds; assuming the thief... The position of time is ,exist The position of time is ;in, and All ~ Within the range, and < ; Based on the surveillance video, the thief was identified. Lateral movement speed (The lateral movement speed is positively measured in the east direction), the thief is in longitudinal movement speed (With north as the positive direction for longitudinal movement speed), construct the thief's position. The first state vector at time : ; Will As a thief Observation vector at time : ; Construct the state transition matrix : ; Construct the observation matrix : ; Construction process noise covariance matrix : ; Select a stationary shelf in the supermarket as a reference shelf; select a point on the reference shelf that can be captured by a camera as a reference point; the reference point is preferably the vertex at the edge of the shelf; Obtaining reference points in continuous surveillance video The pixel positions of the frame image are obtained. ~ ; according to ~ Calculate the variance of the horizontal pixel coordinates of the reference point. variance of vertical pixel coordinates Construct the observation noise covariance matrix of the thief : ; using state vectors As the initial state, the thief is set up in... The posterior covariance matrix at time : matrix The dimension is 4×4, matrix The diagonal elements in the image represent the thief's actions in... ~ Within a given time period, the variances of the horizontal position coordinates, the vertical position coordinates, the horizontal movement speed, and the vertical movement speed are represented by the matrix. All off-diagonal elements in the array are zero; The thief ~ The horizontal and vertical position coordinates, lateral movement speed, and vertical movement speed per second during the time period are input into the ARIMA model to perform secondary prediction of the thief's position. The second state vector is constructed using the horizontal axis position coordinates, vertical position coordinates, horizontal movement speed, and vertical movement speed. : Calculate the thief's Prior prediction of the thief's location at that time : ;in, Represents matrix multiplication; Calculate the thief's Prior covariance matrix at time : Where T represents the transpose of the matrix; Calculate the thief's Observational residuals at time : ; Calculate the thief's Residual covariance at time : ; Calculate the thief's Kalman gain at time : Where -1 represents the inverse of the matrix; Update the thief in Prior predictions at that time yield the optimal estimate. : ; Update the thief in The covariance matrix at time is used to obtain the posterior covariance matrix. : ;in, Represents a 4×4 identity matrix; Using the thief in Optimal estimation at time and state transition matrix Calculate the thief's Prior estimation of position at time : ;in, exist ~ Within the range, and < ; This indicates that the thieves predicted using the ARIMA model are... The position at that time; Get current time Based on the thief's current location, repeatedly calculate the thief's location at that time. The method of prior estimation of location in time uses an ARIMA model in conjunction with Kalman gain to predict the location changes of thieves until it is predicted that the thief has reached a specific merchandise area; where the ARIMA model is used for prediction and the Kalman gain is used for correction.

[0024] It should be noted that the above prediction process is as follows: based on the current time and the thief The location and speed at that time can be used to predict the thief's location. The next second Location Determine the location Is it a location within a specific product section of a supermarket? If so, pause the prediction and move the location... The corresponding merchandise area location is used as the first target area; if not, prediction continues until, after s seconds, the ARIMA model predicts that the thief has arrived at a merchandise area. The thief predicted by the ARIMA model will then be... The shopping area that is reached at a certain time will be the first target area; time Used as the first reference time; The commodity area predicted by the ARIMA model is used as the first target area; the first reference time when the thief arrives at the first target area is recorded. ; Get the number of patrol personnel Divide all products into There are 1 patrol area, and each patrol area is the responsibility of one patrol officer; Identify the patrol area corresponding to the primary target area as the key area; obtain the location of the thief. , distance position The nearest product area is used as the initial target area, and the patrol area corresponding to the initial target area is used as the initial area. Determine whether the initial area and the key area are the same area, and execute different sub-branch patrol strategies accordingly; Sub-branch 21: If the initial region and the key region are the same region, then the region outside the key region is treated as a secondary region, based on the current time. Compared with the first reference time Adjust patrol routes in key areas based on time differences; Sub-branch 211: In the secondary area, based on the real-time location of the patrol personnel, calculate the fusion cost from the patrol personnel's current commodity area to other commodity areas in the patrol area, and use the TSP algorithm to select the path with the minimum total cumulative cost as the patrol route for each patrol personnel in the secondary area. The calculation process for the cumulative cost corresponding to the secondary region is as follows: Let the number of commodity zones in the secondary area be . Let the first to the second minor region be... The risk coefficient for each commodity zone is: ~ ; Let the location of the patrol personnel in the secondary area be... Calculate the position sequentially To the 1st to the The path length of each product area is obtained. ~ ; right ~ and ~ Standardize them separately to eliminate dimensions, and obtain ~ and ~ ; In order ~ As molecules, in sequence with ~ Using the denominator, calculate the first to the second sub-regions. The cost of integrating individual commodity zones, resulting in ~ ; exist ~ The area with the lowest integration cost was selected as the location for patrol personnel. The next patrol target was to be located repeatedly. The next step for patrolling the target is to use the TSP algorithm to traverse all commodity areas in the secondary area to obtain the patrol route for the secondary area. Sub-branch 212: Obtain the location of patrol personnel in key areas and count the number of merchandise areas in key areas. The initial target area is designated as Area C, and the first target area is designated as Area D; the path from Area C to Area D is the primary patrol route, and patrol personnel will patrol at the first reference time. Upon arriving at Area C, patrol all commodity areas within the key patrol route; Obtain the patrol personnel's movement speed and calculate... and time difference ; Count the number of paths in key areas Based on the patrol personnel's movement speed, calculate the number of patrol personnel passing through the key areas from the 1st to the 2nd. The travel time of each path is obtained. ~ ; Will ~ Arrange in ascending order from smallest to largest to obtain ~ ; Compare and The size, if ≥ If the iteration stops, the patrol personnel will... The patrol will start at any merchandise area along the corresponding path, and the patrol speed will be increased to [a certain value]. times; like ≥ Then calculate ~ The former The sum of terms makes ~ The former The sum of the terms is greater than or equal to the sum of the terms for the first time. ; The value range is: 1~ ; exist ~ forward Among the paths corresponding to the items, completely unreachable paths are removed, resulting in barcode path; The value range is: 1~ Completely unreachable path representation: ~ forward In the path corresponding to an item, there is a path that has no shared product area with any other path; exist ~ corresponding In the bar chart path, supplementary paths are introduced to obtain the first patrol route of the patrol personnel; Determine if there is a common product area between the first and second target paths; If so, check if there is a common product area between the second and third target paths; If not, the shortest path from the first marker path to the second marker path is introduced as a parameter path; the travel time of patrol personnel along the parameter path is calculated. ; Compare With the The duration of travel time for each target path; like Less than or equal to the If the path travel time is specific to a target, then a parameter path is introduced, discarding the first path. Individual target path; like Greater than the If the passage time of a single target path is determined, the second target path is discarded. Then, check if there is a common product area between the first and third target paths. Repeat the steps for introducing parameter paths until a target path is found. ~ forward The paths corresponding to the items are organized into reachable paths from any commodity area of ​​the first marked path to area C, thus obtaining the first patrol route; Repeat sub-branch 211, traverse the fusion weight of the remaining commodity areas not patrolled by the first patrol route and the key patrol route in region D, and generate the second patrol route by traversing the remaining areas in region D. Sub-branch 22: If the initial region and the key region are not the same region, then the path from the initial region to the key region is taken as the quasi-path; based on the current time... Compared with the first reference time Adjust patrol routes in key areas based on time differences; The patrol areas traversed by the quasi-path are successively designated as transition areas, and the patrol areas not traversed by the quasi-path are designated as secondary areas. The pathfinding steps of sub-branch 211 are repeated in the secondary areas to formulate the patrol route. In the transition area, the route traversed by the quasi-path is taken as the key patrol route for each transition area. The pathfinding steps of sub-branch 212 are repeated to formulate the patrol route.

[0025] Example 2 Please see Figure 2 The intelligent agent self-recovery execution system based on execution trajectory includes: Video acquisition module: used to acquire surveillance video inside the supermarket; Behavior Analysis Module: Used to define basic behavior chains and construct finite state machines; acquire real-time surveillance video of the supermarket, analyze the behavior of each customer in the video, and obtain multi-dimensional real-time characteristics of each customer; based on the multi-dimensional real-time characteristics of each customer, assign an independent state machine instance to each customer, and determine whether there is any abnormal behavior in the supermarket; if not, skip the behavior chain update module and enter the patrol planning module; if so, extract video clips showing theft behavior from the surveillance video as sample videos. Behavior chain update module: used to preprocess sample videos and output multi-channel features; set up image libraries of different human actions and train a recognition model to recognize human actions; convert the multi-channel features of customers into atomic events of customers through the recognition model; convert atomic events into primary behavior chains, merge some actions in the primary behavior chains to obtain a supplementary chain of the basic behavior chain. Patrol planning module: Used to acquire historical anomaly data in the supermarket, calculate the risk coefficient of each product area in the supermarket; detect whether there is theft in the surveillance video in real time; if not, patrol the product area according to the risk coefficient; if so, immediately alarm and record the location of the thief in real time, and patrol the area where the thief is located.

[0026] The above formulas are all dimensionless calculations. The formulas are derived from software simulations using a large amount of collected data to obtain the most recent real-world results. The preset parameters in the formulas are set by those skilled in the art according to the actual situation. For example, there are weighting coefficients and proportional coefficients. The values ​​set are to quantify each parameter to obtain a specific value, which is convenient for subsequent comparison. The values ​​of the weighting coefficients and proportional coefficients are only required to not affect the proportional relationship between the parameters and the quantified values.

[0027] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A self-recovery execution method for intelligent agents based on execution trajectories, characterized in that, Specific methods include: Define the basic behavior chain and build a state machine; acquire surveillance video, analyze the behavior of people in the surveillance video, and determine whether there is any abnormal behavior in the video; if not, do not process it; if so, extract sample video from the surveillance video. The sample video is preprocessed to output multi-channel features; an image library of different human actions is set up to train a recognition model for recognizing human actions; the multi-channel features are converted into atomic events through the recognition model; the atomic events are converted into primary behavior chains, and some actions in the primary behavior chains are merged to supplement the basic behavior chains; Obtain historical anomaly data and calculate the risk coefficient of each area to be patrolled; detect whether there is any abnormal behavior in the surveillance video; if not, formulate a patrol route according to the risk coefficient; if so, immediately alarm and record the abnormal location, and patrol the area where the abnormal location is located.

2. The agent self-recovery execution method based on execution trajectory according to claim 1, characterized in that, The steps for judging abnormal behavior are as follows: Define the basic behavior chain, and based on the basic behavior chain, define the state and state transition conditions; Match motion clips of each person in the surveillance video, identify each person's actions, and output multi-dimensional real-time features of each person. Based on each person's traceable ID, create an independent state machine instance for each person, and match each person's real-time state while shopping in the supermarket according to the state transition determination rules. Define a normal shopping state transition path as the standard state transition path, and detect the state changes of each person; Scenario 1: If a person has fully experienced the state changes corresponding to the standard state transition path, and the person's state has never been in a warning state or an undefined state, then the person is normal. Scenario 2: If a person has experienced the state change corresponding to the standard state transition path, but the person's state has changed to the alert state, then in the state machine instance corresponding to the person, it is determined whether the person's last state change matches the alert cancellation path. If a match is found, then the person is normal; If there is no match, the person is considered abnormal; Scenario 3: If the change in a person's status does not meet the requirements of Scenario 1 and Scenario 2, then it is determined that the person has exhibited abnormal behavior, and an alarm should be raised immediately. In real time, video segments corresponding to scenarios two and three are extracted from the surveillance video and used as sample videos to fit a new behavior chain.

3. The agent self-recovery execution method based on execution trajectory according to claim 2, characterized in that, The fitting steps for the new behavioral chain are as follows: Extract the bounding box of each person's body from the sample video; based on the bounding box, crop each video frame in the sample video pixel by pixel to obtain the ROI image of each person; Identify skeletal key points in the human pixel images of each person, and perform neutral alignment, size normalization and temporal unification processing on the ROI images of each person. The horizontal pixel coordinates, vertical pixel coordinates, and visibility of the ROI image are concatenated as three independent channels to obtain a standardized sample image tensor. Train a recognition model to identify human actions. Input sample image tensors into the recognition model. Based on the position of each person in the sample video and the actions recognized by the model, output the atomic events of each person in the sample video. Arrange the actions of each person in the atomic event in chronological order to obtain the primary behavior chain.

4. The agent self-recovery execution method based on execution trajectory according to claim 3, characterized in that, The fitting steps for the new behavioral chain also include: Get the number of actions in the template chain ; In the sample video, select behavior chain b; Get the number of actions in behavior chain b Set the threshold for normalized similarity. ; Transform the actions in behavior chain b and template chain into semantic vectors; Based on the number of actions in behavior chain b and template chain, set up a baseline chain and a control chain; The number of identical semantic vectors in the baseline chain and the control chain. Calculate the normalized similarity between behavior chain b and template chain. ,judge Is it greater than or equal to? ; If it is greater than or equal to, it means that the customer's primary behavior chain and basic behavior chain are highly similar. The existing basic behavior chain will be used directly for judgment, and no action will be taken. If the similarity is less than 0, it indicates that the customer's primary behavior chain and basic behavior chain have low similarity. Based on all customer actions in behavior chain b, the encoder stage of the recognition model extracts the corresponding values ​​for each customer action. The diagram shows the merging of actions in behavior chain b.

5. The agent self-recovery execution method based on execution trajectory according to claim 4, characterized in that, The steps to merge the actions in action chain b are as follows: In behavior chain b, select two temporally consecutive ROI plots, denoted as plot [plot name missing]. and picture ; Get Image Previous frame ; Constructing a graph The three-dimensional features include: spatial features, action features, and interaction features. Get Image The first two and first four frames of images are used to obtain the image. and picture ; Get Image and picture The three-dimensional features, in the figure To the map Then to the picture Based on the time span, calculate the graph Curvature corresponding to spatial features motion characteristic curvature curvature of interactive features ; Calculation graph The curvature corresponding to spatial features, action features, and interaction features is obtained. , and ; Set the curvature determination error, and judge sequentially. and , and as well as and Are all the differences within the curvature determination error? If so, then retain the graph in behavior chain b. Corresponding atomic events, delete graph The corresponding atomic events; if none or only some are present, then the graph is preserved in behavior chain b. and picture The corresponding atomic events.

6. The agent self-recovery execution method based on execution trajectory according to claim 1, characterized in that, The steps for planning patrol routes are as follows: Calculate the risk coefficient for each area to be patrolled based on historical anomaly data; The risk coefficients of each area to be patrolled are arranged in ascending order from smallest to largest. The threshold for judging the risk coefficient of the commodity area is extracted to obtain the threshold. With threshold ; Risk coefficient greater than or equal to the threshold The area to be patrolled was designated as a high-risk area. Risk coefficient less than or equal to the threshold The area to be patrolled is designated as a low-risk area; The remaining areas to be patrolled are designated as medium-risk areas; Detect any unusual behavior in surveillance videos; If not, then patrol the commodity areas corresponding to high-risk, medium-risk, and low-risk areas in sequence according to the size of the risk coefficient; If so, record the abnormal location, determine whether the abnormal location has changed, and formulate different branch patrol plans; Branch 1: If the abnormal location remains unchanged, use reverse update to determine the patrol route; Branch 2: Abnormal location changes, increase the number of patrol personnel, predict abnormal location changes, and formulate patrol routes.

7. The agent self-recovery execution method based on execution trajectory according to claim 6, characterized in that, The steps for branch one are as follows: Obtain the location of patrol personnel and abnormal locations, and record them as locations. With position ; Obtain the location of each area to be patrolled, and record it as the location. ~location Establish a data stack and store the locations of all areas to be patrolled into the stack; Calculate the position sequentially Arrive at the location ~location The route length is obtained. ~ ;extract ~ minimum value ,Will The corresponding area to be patrolled is designated as the first patrol zone; Remove the location of the first patrol zone from the data stack; The areas outside the first patrol zone are designated as secondary patrol zones; the route length from the first patrol zone to each secondary patrol zone is calculated sequentially. ~ ;extract ~ minimum value ,Will The corresponding secondary patrol area is moved out of the data stack as the second patrol area; Repeat the reverse update steps for finding the first and second patrol zones, and search for the third to fourth patrol zones in the remaining unpatched zones. Patrol the area until the data stack is empty; Record the pop order of each area to be patrolled in the data stack, arrange all areas to be patrolled in the pop order to obtain the primary patrol route; reverse the primary patrol route to obtain the patrol route.

8. The agent self-recovery execution method based on execution trajectory according to claim 7, characterized in that, The steps for processing branch two are as follows: Get current time Based on the current time of the anomaly's location, predict its positional changes until it is predicted to reach the patrol area, thus obtaining the first target area; record the first reference time when the anomaly moves to the first target area. ; Identify the patrol area corresponding to the primary target area as the key area; acquire anomaly locations. , distance position The nearest product area is used as the initial target area, and the patrol area corresponding to the initial target area is used as the initial area. Determine whether the initial area and the key area are the same area, and execute different sub-branch patrol strategies accordingly; Sub-branch 21: If the initial region and the key region are the same region, then the region outside the key region is treated as a secondary region, based on the current time. Compared with the first reference time Adjust patrol routes in key areas based on time differences; Sub-branch 211: The path from the initial target area to the first target area is designated as the primary patrol route, and patrol personnel will patrol at the first reference time. Upon reaching the initial target area, patrol the key areas along the designated patrol route; Sub-branch 212: Calculate the time cost of each path in the key patrol area and set the patrol route based on the time cost; Sub-branch 22: If the initial region and the key region are not the same region, then the path from the initial region to the key region is taken as the quasi-path; based on the current time... Compared with the first reference time Adjust patrol routes in key areas based on time differences.

9. The intelligent agent self-recovery execution method and system based on execution trajectory according to claim 8, characterized in that, The processing steps for sub-branch 212 are as follows: Sub-branch 212: Obtain the location of patrol personnel in key areas and count the number of merchandise areas in key areas. The initial target area is designated as region C, and the first target area is designated as region D. Obtain the movement speed of the patrol personnel and calculate... and time difference ; Count of paths in key areas Based on the patrol personnel's movement speed, calculate the number of patrol personnel passing through the key areas from the 1st to the 2nd. The travel time of each path is obtained. ~ ; Will ~ Arrange in ascending order from smallest to largest to obtain ~ ; Compare and The size, if ≥ Then stop iterating, and Any product area along the corresponding path will serve as the starting point for patrols, and the patrol speed will be increased. like ≥ Then calculate ~ The former The sum of terms makes ~ The former The sum of the terms is greater than or equal to the sum of the terms for the first time. ; exist ~ forward Among the paths corresponding to the items, completely unreachable paths are removed, resulting in Barmark path; In the marked path, a supplementary path is introduced to obtain the first patrol route of the patrol personnel; Determine if there is a common area between the first and second target paths; If present, do not process. If not, then the shortest path from the first target path to the second target path is introduced as the parameter path; Calculate the travel time of patrol personnel along the parameter path. ; judge Is it less than or equal to the first? Travel time for individual routes; If so, then introduce the parameter path and discard the first one. Individual target path; If not, then discard the second target path; Repeat the steps of introducing the parameter path until the first patrol route is obtained.

10. An agent self-recovery execution system based on execution trajectory, applicable to any one of the agent self-recovery execution methods based on execution trajectory according to claims 1-9, characterized in that, The system includes: Video acquisition module: used to acquire surveillance video inside the supermarket; Behavior Analysis Module: Used to define basic behavior chains, analyze the behavior of people in surveillance videos, and obtain multi-dimensional real-time characteristics of each person; based on the multi-dimensional real-time characteristics of each person, determine whether there is any abnormal behavior in the video; if not, skip the behavior chain update module; if so, extract video clips of abnormal behavior from the surveillance video as sample videos. Behavior chain update module: used to preprocess sample videos and output multi-channel features; train a recognition model to recognize human actions, and convert the multi-channel features of people into atomic events through the recognition model; convert atomic events into primary behavior chains, merge some actions in the primary behavior chains, and supplement the basic behavior chain; Patrol planning module: Used to acquire historical anomaly data, calculate the risk coefficient of each area to be patrolled; detect whether there is abnormal behavior in the surveillance video; if not, patrol the area to be patrolled according to the risk coefficient; if so, immediately alarm and record the abnormal location in real time, and patrol the area where the abnormal location is located.

Images