A method and system for identifying theft in a shelving environment

Abstract

The application provides a kind of theft behavior identification method and system in shelf scene, belong to action recognition technical field, this method includes: obtaining the monitoring video stream in shelf scene;The human key point information of each image frame in monitoring video stream is obtained by key point detection algorithm, and at least one image frame subsequence is obtained by non-overlapping segmentation monitoring video stream;Based on the human key point information of the image frame contained in each image frame subsequence, the corresponding action feature representation is obtained;Determine the semantic similarity between each standard feature representation in the standard feature representation-word mapping relationship of each action feature representation, and determine the corresponding word of each action feature representation based on semantic similarity, to obtain the word sequence corresponding to the monitoring video stream of each action feature representation;Word sequence is input into semantic classification model, and the theft behavior identification result of monitoring video stream is output to obtain.The application can realize high reliability normal action recognition and theft action capture in shelf scene.

Description

Technical Field

[0001] This invention relates to the field of motion recognition technology, and in particular to a method and system for recognizing theft behavior in a shelf setting. Background Technology

[0002] Currently, methods for identifying shelf theft mainly fall into several categories: one is anti-theft technology based on Radio Frequency Identification (RFID). This involves deploying RFID detection devices in areas such as shelf areas, checkout areas, and exits, and periodically analyzing the consistency between the amount of goods removed, the amount of goods paid for, and the amount of goods taken out of the store to determine if unpaid theft has occurred. This was primarily used as an early anti-theft solution in unmanned stores and smart vending machines. However, this solution has drawbacks such as limited applicability and a lack of real-time capability. With the rapid development of computer vision technology and the improvement of computing power in monitoring equipment, real-time identification of theft based on visual analysis is gradually gaining widespread application.

[0003] Current methods for identifying theft behavior in shelf scenarios based on visual analysis technology mainly fall into two categories: one type uses anti-theft technology based on single-modal video target detection, which can identify targets such as shelves, goods, or people in video frames through target detection algorithms, and comprehensively judge theft behavior based on target detection information such as changes in the position of goods in different frames or key frames of single actions. However, this approach has weak anti-interference capabilities and limitations in shelf scenarios. For example, key actions can be obscured in a panoramic monitoring view, leading to a high false detection rate, and blind spots in local monitoring views can easily cause missed detection of goods. Therefore, another type of method, based on graph convolution or deep learning neural network action recognition models for identifying theft actions, has gradually become the mainstream identification scheme. However, after analysis and verification, the inventors of this application believe that the actions identified by this type of action recognition model require a complete behavioral process (i.e., the video stream to be identified has a certain duration) and require a clearly categorized action video segment as input. This results in poor adaptability of existing action recognition models to action recognition in shelf scenarios because: ① The length of the video stream to be identified in the shelf scene is difficult to determine, and it is impossible to accurately extract the video stream of a certain duration from the monitoring video: The interaction time between customers and products in the shelf scene varies greatly, ranging from only 2-3 seconds to 10-20 minutes, resulting in a large range of variation in the length of the video stream to be identified. Existing action recognition models usually rely on image sequences of stable length to identify actions during the training and inference stages, which is difficult to adapt to the uncertain duration of the video stream to be identified in the shelf scene. ② The interaction between customers and goods in shelf scenarios is quite complex: Existing action recognition models are mainly designed to identify action patterns with obvious differences. However, in shelf scenarios, at certain times, the differences between normal (non-theft) and theft actions of customers interacting with goods are not significant, or normal shopping actions and theft actions may alternate (e.g., the action in the previous image frame is similar to theft, but the action in the next image frame is putting the goods back in their original position). This requires the action recognition model to have a fine ability to distinguish subtle human movements, but existing action recognition models are difficult to achieve similar detection accuracy. Furthermore, the interaction between customers and goods in shelf scenarios is irregular, and there may be situations where the behaviors contained in video clips are too complex (e.g., containing many taking down and putting back, etc.), causing the action recognition model to be unable to summarize the overall behavior.

[0004] Therefore, how to identify theft actions in video streams by combining behavioral characteristics of shelf scenarios is an urgent problem to be solved. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a method and system for identifying theft behavior in a shelf setting, which can overcome the poor adaptability of existing action recognition models in shelf settings and achieve accurate differentiation between normal actions and theft behavior.

[0006] One aspect of the present invention provides a method for identifying theft behavior in a shelf setting, the method comprising the following steps: Acquire the surveillance video stream in the shelf scene; wherein, the surveillance video stream includes multiple image frames containing human images; Human key point information of each image frame in the surveillance video stream is obtained by key point detection algorithm, and the surveillance video stream is segmented by segmentation algorithm to obtain at least one image frame subsequence; wherein, there are no overlapping image frames in any two image frame subsequences. Using an action feature extraction algorithm, the action feature representation corresponding to each image frame subsequence is obtained based on the human key point information of the image frames contained in each image frame subsequence; For each image frame subsequence, the semantic similarity between the action feature representation and each standard feature representation in the pre-stored standard feature representation-term mapping relationship is determined, and the term corresponding to the action feature representation is selected from the mapping relationship based on the semantic similarity; wherein, a standard feature representation corresponds to a meta-action of theft, and the meta-actions corresponding to all standard feature representations in the mapping relationship include at least all meta-actions of theft. Based on the temporal sequence of each image frame subsequence in the surveillance video stream, the corresponding word sequence of the surveillance video stream is obtained. The word sequence is then input into a pre-trained semantic classification model, and the output is the identification result of whether the surveillance video stream contains theft behavior.

[0007] In some embodiments of the present invention, a segmentation algorithm is used to segment the surveillance video stream to obtain at least one image frame subsequence, including: The surveillance video stream is uniformly segmented, or non-uniformly segmented based on human key point information, to obtain at least one image frame subsequence. Among these, uneven segmentation of the surveillance video stream based on human key point information includes: The motion displacement of each image frame is determined based on information from specific human body key points in each image frame; among which, specific human body key points include elbow key points and wrist key points. The motion trajectory waveform is formed based on the motion displacement of all image frames in the monitoring video stream, and the monitoring video stream is segmented based on the extreme points in the motion trajectory waveform.

[0008] In some embodiments of the present invention, after segmenting the surveillance video stream based on extreme points, the method further includes: Determine the number of frames in each initial sequence obtained by segmenting the surveillance video stream based on extreme points; For each initial sequence, if the number of frames in the initial sequence does not meet the set frame count condition, the initial sequence is subjected to frame extraction, resampling, and merging processing based on a preset image sequence processing strategy. The image sequence processing strategies include: If the number of frames in the initial sequence is less than the first set frame number threshold, then the initial sequence and the initial sequence following the initial sequence in the timing sequence are merged. If the number of frames in the initial sequence is between the first set frame number threshold and the second set frame number threshold, then the initial sequence is resampled. If the number of frames in the initial sequence is greater than the second set frame number threshold, then the initial sequence is subjected to frame extraction.

[0009] In some embodiments of the present invention, the motion feature extraction algorithm is the PoseC3D algorithm, and the human body key point information includes key point coordinates and key point confidence. Using action feature extraction algorithms, action feature representations corresponding to each image frame subsequence are obtained based on the human keypoint information of the image frames contained in each image frame subsequence, including: For each image frame in each image frame subsequence, the information of each human key point is mapped to a heat map according to the confidence of each key point in the image frame. The heat map obtained by stacking all human key points in the image frame is used to form the heat matrix corresponding to the image frame. For each image frame subsequence, the heat matrix corresponding to each image frame is stacked according to the temporal order of each image frame in the image frame subsequence to obtain the heat matrix corresponding to the image frame subsequence; The PoseC3D algorithm inputs the heatmap matrix corresponding to each image frame subsequence and outputs the action feature representation corresponding to each image frame subsequence.

[0010] In some embodiments of the present invention, the PoseC3D algorithm is input based on the heatmap matrix corresponding to each image frame sub-sequence, including: Input the heatmap corresponding to each image frame subsequence into the PoseC3D algorithm; or input the heatmap corresponding to each image frame subsequence and the RGB image sequence corresponding to each image frame subsequence into the PoseC3D algorithm.

[0011] In some embodiments of the present invention, determining the semantic similarity between the action feature representation and each standard feature representation in the pre-stored standard feature representation-lexical mapping relationship, and selecting the lexical corresponding to the action feature representation from the mapping relationship based on the semantic similarity, includes: Determine the Euclidean distance between the action feature representation and each standard feature representation in the pre-stored standard feature representation-lexical mapping relationship, and determine the semantic similarity between the action feature representation and each standard feature representation based on the Euclidean distance; Select the lexical corresponding to the standard feature representation with the highest semantic similarity from the pre-stored standard feature representation-lexical mapping relationship, and use it as the lexical corresponding to the action feature representation.

[0012] In some embodiments of the present invention, the standard feature representation-lexical mapping relationship is obtained in the following manner: Acquire composite surveillance video streams in a shelf setting; wherein, the composite surveillance video stream is a collection of surveillance video streams including multiple shelf areas, multiple personnel targets, and / or multiple identifiable theft actions; By using keypoint detection algorithm, segmentation algorithm and action feature extraction algorithm, the action feature representations corresponding to each image frame subsequence in the composite surveillance video stream are obtained; Using clustering algorithms, the action feature representations corresponding to all image frame subsequences in a composite surveillance video stream are clustered to obtain multiple clusters. The cluster centers of each cluster are used as standard feature representations, and the standard feature representation-lexical mapping relationship is constructed by labeling the standard feature representations with lexical units.

[0013] In some embodiments of the present invention, acquiring the monitoring video stream in a shelf scenario includes: For shelf scenarios, acquire raw monitoring video streams containing multiple image frames; The location of goods and / or people in each image frame of the original surveillance video stream is determined by the target detection algorithm. The starting and ending image frames of the surveillance video stream are determined based on the target location of the goods and / or the target location of the personnel, thereby extracting the surveillance video stream from the original surveillance video stream.

[0014] In some embodiments of the present invention, the pre-trained semantic classification model is constructed based on Transformer; the pre-trained semantic classification model is obtained in the following manner: Obtain the training word sequence corresponding to the training sample; The training word sequence is input into the semantic classification model to be trained, and the model parameters are iteratively adjusted based on the model output to obtain the pre-trained semantic classification model.

[0015] Another aspect of the present invention provides a theft behavior recognition system in a shelf setting, including a processor, a memory, and a computer program / instructions stored in the memory. The processor is used to execute the computer program / instructions, and when the computer program / instructions are executed, the system implements the steps of the method described in any of the above embodiments.

[0016] The theft behavior recognition method and system proposed in this invention for shelf scenes avoids the problem of limited video stream length by dividing a continuous video stream into multiple continuous image frame subsequences. Theoretically, it can handle shelf scene video streams of unlimited length. Furthermore, this application also designs a lexical mapping mechanism, which pre-generates a lexical library with distinguishable elements based on the action characteristics of the shelf scene. This allows the subtle differences between various normal actions and theft actions to be accurately expressed by lexical units, overcoming the defect of existing action recognition models that cannot clearly classify complex action sequences. Thus, it achieves highly reliable recognition of normal shopping behavior and capture of theft behavior in shelf scenes.

[0017] Additional advantages, objects, and features of the invention will be set forth in part in the description which follows, and will also become apparent in part to those skilled in the art upon studying the description, or may be learned by practice of the invention. The objects and other advantages of the invention can be realized and obtained by means of the structures specifically pointed out in the description and drawings.

[0018] Those skilled in the art will understand that the objectives and advantages achievable with the present invention are not limited to those specifically described above, and that the above and other objectives achievable with the present invention will become clearer from the following detailed description. Attached Figure Description

[0019] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, are not intended to limit the scope of the invention. In the drawings: Figure 1 This is a flowchart illustrating a method for identifying theft behavior in a shelf setting according to an embodiment of the present invention.

[0020] Figure 2 This is a flowchart illustrating a theft detection method in a shelf scenario according to another embodiment of the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the illustrative embodiments and descriptions of this invention are used to explain the invention, but are not intended to limit the invention.

[0022] It should also be noted that, in order to avoid obscuring the invention with unnecessary details, only the structures and / or processing steps closely related to the solution according to the invention are shown in the accompanying drawings, while other details that are not closely related to the invention are omitted.

[0023] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, element, step, or component, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0024] In the following description, embodiments of the invention will be illustrated with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar parts, or the same or similar steps.

[0025] Applying existing motion recognition solutions to identify theft in shelf scenarios (such as supermarket shelves) can not fully and accurately express the target's behavior at any given moment, even if visual analysis can identify suspected theft in a single image frame or within a certain time period. This is due to the complex factors such as occlusion in shelf scenarios, low hand resolution, overlapping of multiple people, and extremely subtle differences between normal shopping actions and theft actions.

[0026] Existing computer vision-based action recognition technologies have the following drawbacks: ① Existing solutions cannot accurately distinguish between various normal shopping actions and theft actions with extremely subtle differences in their nature, such as the normal "taking and putting back" versus theft as "taking and hiding." Therefore, in shelf scenarios, inaccurate judgment of the timing of actions and missing theft behavior recognition lead to high false positive and false negative rates. Moreover, existing solutions are almost unable to accurately identify theft behaviors formed by complex and diverse combinations of actions. ② Existing technologies focus too much on action analysis at a specific moment or time period and are limited by the duration of video analysis, neglecting the patterns of complete long-term behavior of individuals.

[0027] To address the core shortcomings of existing shelf anti-theft technologies in terms of accuracy, scenario adaptability, and practicality, this application proposes a novel theft action recognition method based on semantic understanding. This method effectively segments the complete interactive video stream between personnel targets (such as customers, stock clerks, and replenishment personnel) and product targets, and effectively extracts behavioral features from the segmented image frame subsequences to form a temporally sequential action feature representation sequence. The action feature representation sequence is then transformed into a simple word sequence, thereby discretizing the continuous action signal into a discrete symbol sequence that can be processed by a semantic classification model (such as Transformer). Furthermore, by parsing the semantic and temporal logic of the word sequence, the monitoring video stream is analyzed, enabling accurate identification of theft behavior in shelf scenarios and capturing human behavioral intentions.

[0028] This application innovatively combines video stream segmentation, extraction of temporal feature sequences, BytePair Encoding (BPE) clustering, and semantic discretization to overcome the limitations of traditional single visual features. By segmenting the video stream and focusing on extracting behavioral features within a short timeframe, it avoids the loss of key behavioral features caused by long time periods and frame sampling. Furthermore, for the temporal feature sequences formed after segmentation, each feature can be labeled as a meta-action word obtained through the BPE clustering strategy. This process transforms continuous and ambiguous action semantics into discrete meta-action words, laying the foundation for subsequent encoding. A semantic classification model is used to perform temporal reasoning on token encoding, realizing a complete link from "identifying action features" to "understanding action logic." It should be noted that this application aims to improve the technological level in the field of public safety, specifically improving the accuracy of theft behavior identification. Its application will help achieve legally mandated public safety goals more efficiently and systematically, thus having positive social benefits.

[0029] Figure 1 This is a flowchart illustrating the theft detection method for shelf-based scenarios proposed in this application. Figure 1As shown, the method includes steps S110 to S150, as detailed below: Step S110: Acquire the monitoring video stream of the shelf scene. The shelf scene is a digital management scenario that mainly focuses on product display, inventory status, and customer interaction.

[0030] The surveillance video stream obtained in this application is extracted from the original surveillance video stream (containing multiple image frames) captured by a camera. The length of the surveillance video stream can be customized. For example, this application designs a method to determine the start and end image frames of the surveillance video stream based on the location of the product target and / or the location of the personnel target (the positions of the product target and personnel target can be determined by a target detection algorithm), thereby extracting the surveillance video stream from the original surveillance video stream. At this point, the surveillance video stream is a sequence of image frames showing a complete interaction between a personnel target and a product target; that is, the surveillance video stream includes multiple image frames containing human images. This application is applicable to the recognition of a person's complete behavior over a long period of time, and is also applicable to the recognition of a person's simple actions over a short period of time.

[0031] As an example, when the target detection algorithm determines that the state of a certain product target has changed (such as a change in the position of the product target), image frame capture for that product target can be initiated (i.e., the image frame where the product target begins to change is used as the starting frame of the monitoring video stream), and the image frame capture can be terminated when the continuous change in the state of the product target ends (i.e., the state of the product target does not change within a set time period) (i.e., the image frame where the continuous change in the state of the product target ends is used as the ending frame of the monitoring video stream); when the target detection algorithm determines that there is a person target in front of the shelf (such as the distance between the target person and the shelf is less than a set distance threshold and the target person's stay time is greater than a set time threshold), image frame capture can be initiated, and the image frame capture can be terminated when the target person leaves the shelf (i.e., the distance between the target person and the shelf is less than a set distance threshold and the target person's stay time is greater than a set time threshold). Image frame capture can be terminated when the distance between the product target and the person target is not less than a set distance threshold. This application can also be designed to start image frame capture when the start condition for achieving interaction between the product target and the person target is determined, and end image frame capture when there is no interaction. If the correlation condition of whether the product target and the person target interact is used as the basis for the start of capture, image frame capture can be started when the target detection algorithm determines that the state of the product target has changed and there is a person target in front of the shelf, or when the target detection algorithm determines that the distance between the product target and the person target is less than a set interval threshold, and can end image frame capture when the state of the product target continues to change and the person target leaves the shelf, or end image frame capture when the distance between the product target and the person target is not less than a set interval threshold.

[0032] Existing target detection-based schemes rely solely on product trajectories or single action features, lacking interaction modeling between humans and products. Therefore, to facilitate the subsequent step S120 for human key point recognition, this application can incorporate changes in the position of the human target in the initial data extraction criteria.

[0033] In this application, the surveillance video stream to be identified is not limited by the duration of the actual interaction between the person and the product. Theoretically, the behavior of the person in a shelf setting does not possess general regularity or rationality. This means that in complex shelf environments, the duration of the interaction between the person and the product can vary. For example, when customers shop, they may engage in multiple invalid and repetitive actions such as taking items and putting them back on the shelf. Therefore, to overcome the shortcomings of existing video streams with limited length that lead to inaccurate identification, this application interprets all the behavior of the person in front of the shelf during the time period as the complete interaction between the person and the product.

[0034] Step S120: Obtain human key point information in each image frame of the surveillance video stream using key point detection algorithms (such as traditional SIFT (Scale-Invariant Feature Transform), OpenPose, deep learning-based methods, TransPose, or Transformer, etc.). Human key point information is the key point information of the human target in the image frame. The surveillance video stream is segmented using a segmentation algorithm to obtain at least one image frame subsequence (the image frame subsequence can also be called a video stream segment).

[0035] The obtained human keypoint information for each image frame may include the image frame number, keypoint type, keypoint coordinates (specifically, the pixel coordinates of the keypoint within the image frame), and keypoint confidence. The keypoint type can be one of the 17 keypoint types in the COCO standard, or it can be a keypoint in other formats; this invention is not limited to these. Keypoint detection can be performed by monitoring each image frame in the video stream as input, or by using the image within the bounding box of the person target in each image frame as input; this invention is not limited to these methods.

[0036] As an example, to demonstrate the interaction between personnel and product targets, this application can also utilize a keypoint detection algorithm to detect product keypoint information in each image frame of the surveillance video stream. Product keypoints can be specifically defined according to the type of product in the shelf scene, such as using the center point of the product target as the product keypoint; this invention does not specifically limit this definition.

[0037] Considering that a person's presence in front of the shelf may involve various behaviors or events throughout the entire time, this application simplifies motion feature extraction by dividing the surveillance video stream into multiple video stream segments before executing step S130. This application does not specifically limit the method for segmenting the surveillance video stream; uniform segmentation algorithms or non-uniform segmentation algorithms can be used, as long as the surveillance video stream can be broken down into the smallest granularity, definable, and identifiable basic actions (image frame subsequences can also be called meta-action segments). Examples of different segmentation methods will be given below. Those skilled in the art can also use other segmentation methods besides those described in this application; this invention is not limited thereto.

[0038] For uniform segmentation algorithms, the surveillance video stream can be uniformly segmented based on a preset number of segmentation frames. For example, to ensure proper alignment and facilitate subsequent motion feature extraction, the preset number of segmentation frames can be limited to 48 frames. In this case, the number of frames in the image frame subsequence containing the last image frame of the surveillance video stream may not be equal to the preset number of segmentation frames. This can be achieved by resampling to increase the number of frames in the image frame subsequence to 48, ensuring that the multiple image frame subsequences obtained from uniform segmentation have the same number of frames. This application does not specifically limit the size of the preset number of segmentation frames; it can be set according to requirements.

[0039] For uneven segmentation algorithms, surveillance video streams can be unevenly segmented based on human key point information. For example, the motion displacement of each image frame can be determined based on the information of specific human key points (specific human key points may include four key points such as the left and right elbows and the left and right wrists, and may also include key points with the center of the torso as a reference); a motion trajectory waveform is formed based on the motion displacement of all image frames in the surveillance video stream, and the surveillance video stream is segmented based on the extreme points (maximum points and / or minimum points) in the motion trajectory waveform.

[0040] As an example, the process of unevenly segmenting a surveillance video stream based on human key point information can be as follows: Select four upper limb key points (left and right elbows, left and right wrists), and use the center of the torso as a reference point. Calculate the Euclidean centripetal distance of each upper limb key point pixel coordinate relative to the reference point in each image frame. For each image frame in the surveillance video stream, perform a weighted sum of the four calculated Euclidean centripetal distance values ​​(e.g., based on the movement patterns of humans in a shelf scene, elbow movement is more stable than wrist movement, so the weight of the elbow key point can be set greater than the weight of the wrist key point to reduce noise interference from wrist movement on the weighted summation result; assuming the elbow weight is 2 and the wrist weight is 1), to obtain the weighted centripetal distance of each image frame, and use the obtained weighted centripetal distance as the movement displacement. With the movement displacement as the vertical axis and the temporal sequence of the image frame in the surveillance video stream as the horizontal axis, form a movement trajectory waveform based on the calculated movement displacements of all image frames. Select all the minimum points in the movement trajectory waveform (refer to...). Figure 2 The complete image frame sequence (i.e., the monitoring video stream) obtained in step S110 is segmented using the minimum point as the segmentation point, thereby dividing the monitoring video stream into video stream segments with a relatively short time length. Alternatively, all the maximum points in the motion trajectory waveform can be selected and the monitoring video stream can be segmented using the maximum points as segmentation points. Or, the minimum and maximum points in the motion trajectory waveform can be used as segmentation points to segment the monitoring video stream. In this case, the image frame between the minimum point and a maximum point adjacent to the minimum point can be segmented into an image frame subsequence (except for the first and last image frame subsequences).

[0041] The above method of segmenting the monitoring video stream based on weighted centripetal distance is only an example. Other methods of unevenly segmenting the monitoring video stream based on human key point information can also be used. For example, product target key points can be introduced, and the interaction behavior between product target and human target can be determined based on the distance between human key points and product target key points, thereby realizing uneven segmentation of the monitoring video stream. The specific calculation process can be designed by oneself, and this application will not describe it in detail here.

[0042] This application typically employs a non-overlapping segmentation scheme when segmenting surveillance video streams. This means that no two resulting image frame subsequences overlap. Therefore, when segmenting a surveillance video stream by extreme points, the image frame corresponding to the extreme point does not simultaneously exist in adjacent image frame subsequences. For example, the starting image frame of each image frame subsequence can be set to the image frame corresponding to the extreme point (except for the first image frame subsequence, in which case the starting image frame of the surveillance video stream is used as the starting image frame of the first image frame subsequence, and the image frame preceding the image frame corresponding to the first extreme point is used as the ending image frame of the first image frame subsequence). Alternatively, the ending image frame of each image frame subsequence can be set to the image frame corresponding to the extreme point (except for the last image frame subsequence).

[0043] It should be noted that image frame subsequences used for action recognition are required to possess short-term stationarity and local consistency. Segmentation can decompose continuous actions into the finest-grained, semantically simple, and structurally stable basic units within a specific time window, thus adapting to temporal modeling, feature extraction, and action combination reasoning. Furthermore, the segmentation algorithm must ensure appropriate segmentation granularity, avoiding segmentation with large time frame intervals, and assigning meta-action definitions to the segmented video stream segments.

[0044] Furthermore, to facilitate subsequent action feature extraction, this application also designs a method to sequentially examine the image frame sequence obtained by directly segmenting the surveillance video stream, and then processes it using an image sequence processing strategy. For ease of distinction, this application refers to the image frame sequence obtained by directly segmenting the surveillance video stream as the initial image frame subsequence (hereinafter referred to as the initial sequence). Combining all the image frames of the initial sequence yields the surveillance video stream, and the sequence obtained after processing the initial sequence using the image sequence processing strategy is referred to as the image frame subsequence.

[0045] Image sequence processing strategies can be set based on the number of frames in the image frame sequence (the total number of image frames contained in the image frame sequence). For example, this application can be designed to perform frame extraction, resampling, and merging processing on the initial sequence based on a preset image sequence processing strategy when the number of frames in an initial sequence does not meet the set frame number condition. This application does not specifically set the set frame number condition; the set frame number condition can be a specific frame value or a frame number range. For example, if the subsequent motion feature extraction algorithm is the PoseC3D (Pose Convolutional 3D) algorithm, the set frame number condition can be 48 frames.

[0046] As an example, image sequence processing strategies may include: ① If the number of frames in the initial sequence is less than a first set frame number threshold, then the initial sequence and subsequent initial sequences are merged; ② If the number of frames in the initial sequence is between the first set frame number threshold and the second set frame number threshold (here, the first set frame number threshold ≤ the number of frames in the initial sequence < the second set frame number threshold), then the initial sequence is resampled (e.g., using linear interpolation) to reach the set frame number condition; ③ If the number of frames in the initial sequence is greater than the second set frame number threshold, then the initial sequence is skipped to reach the set frame number condition. Here, the first set frame number threshold < the second set frame number threshold, and the settings of the first and second set frame number thresholds can be designed according to requirements, or they can be used without referring to the size of the set frame number condition. For example, if the surveillance video stream can be divided into three initial sequences with frame numbers of 12, 12 and 45 respectively, the first and second initial sequences can be merged. In this case, the merged sequence with frame number 24 is between the first set frame number threshold 24 and the second set frame number threshold 50. The merged sequence and the third initial sequence are then resampled to meet the set frame number conditions.

[0047] If the number of frames in the merged sequence is still less than the first set frame number threshold, the merged sequence is regarded as an initial sequence and processed again using rules ①, ② and ③ in the above image sequence processing strategy until the set frame number condition is reached; if the number of frames in the initial sequence is less than the first set frame number threshold and there is no next initial sequence to be merged, the initial sequence can be resampled to reach the set frame number condition.

[0048] In some embodiments of the present invention, the image sequence processing strategy may also be as follows: If the number of frames in the current initial sequence is less than a set frame number condition, then if the number of frames in the next initial sequence is greater than a specific difference (the difference between the number of frames in the current initial sequence and the set frame number condition), the current initial sequence is resampled to meet the set frame number condition; if the number of frames in the next initial sequence is not greater than the specific difference, the current initial sequence and the next initial sequence are merged; if the number of frames in the current initial sequence is greater than the set frame number condition, the current initial sequence is frame-stripped to meet the set frame number condition. If the number of frames in the merged sequence is still less than the set frame number condition, processing continues based on the aforementioned rules, which will not be elaborated upon here.

[0049] The two image sequence processing strategies mentioned above are merely examples. Those skilled in the art can design other processing strategies, and this invention is not limited thereto. Furthermore, the human keypoint information in the resampled image frames can be determined based on the human keypoint information in existing image frames. For example, when using linear interpolation for resampling, the temporal data of skeletal keypoints in the initial sequence can be used as a basis. The mapping index and weight of the interpolation frame position can be calculated according to the interpolation step size, and the keypoint pixel coordinates and keypoint confidence of adjacent frames can be linearly fused to ensure the continuity of the temporal features of the video stream segments.

[0050] Step S130: Using an action feature extraction algorithm, the feature representations corresponding to each image frame sub-sequence are obtained based on the human keypoint information of the image frames contained in each image frame sub-sequence. The feature representations obtained in Step S130 are specifically pose feature representations or action feature representations. By combining the action feature representations corresponding to each image frame sub-sequence according to the temporal order of the image frame sub-sequences in the monitoring video stream, the action feature representation sequence corresponding to the monitoring video stream can be obtained.

[0051] This application does not specifically limit the type of action feature extraction algorithm. For example, the action feature extraction algorithm could be the PoseC3D algorithm or the BlockGCN (Block-based Graph Convolutional Network) algorithm, among other skeletal action recognition algorithms. PoseC3D is a 3D-CNN-based skeletal behavior recognition framework that achieves both high accuracy and efficiency, reaching state-of-the-art (SOTA) performance on multiple skeletal behavior datasets. Unlike traditional Graph Convolutional Network (GCN) methods based on the human 3D skeleton, PoseC3D achieves better recognition results using only a stack of 2D human skeleton heatmaps as input.

[0052] In some embodiments of the present invention, when the action feature extraction algorithm is the PoseC3D algorithm, the action feature extraction algorithm is used to obtain the action feature representation corresponding to each image frame subsequence based on the human key point information of the image frames contained in each image frame subsequence. This includes: for each image frame in each image frame subsequence, mapping the information of each human key point (specifically, the pixel coordinates of the key point) to a heatmap according to the confidence level of each key point in the image frame, and forming a heatmap matrix corresponding to the image frame by stacking the heatmaps obtained by mapping all human key points in the image frame; for each image frame subsequence, stacking the heatmap matrix corresponding to each image frame according to the temporal order of each image frame in the image frame subsequence to obtain the heatmap matrix corresponding to the image frame subsequence; inputting the heatmap matrix corresponding to each image frame subsequence into the PoseC3D algorithm, and outputting the action feature representation corresponding to each image frame subsequence.

[0053] As an example, the keypoint information of the multiple video stream segments obtained from the segmentation is processed into the input format data of the PoseC3D algorithm. The process is as follows: By analyzing the human keypoint information of multiple consecutive frames in each image frame subsequence, the human keypoint information of each image frame is mapped point by point to 64 according to the keypoint confidence. In the 64 heatmaps, each human key point in each image frame corresponds to a heatmap; stacking the heatmaps corresponding to all human key points in each image frame forms a 17-heatmap. 64 A 64-dimensional heatmap (assuming keypoint detection is performed according to the COCO format), and a 17-dimensional heatmap. 64 Each of the 64 heat matrices corresponds to one image frame; heat matrices created from all image frames in a stacked image frame subsequence are stacked sequentially to form a 1-dimensional heat matrix. 48 17 64 A 2D human skeleton heatmap of 64 pixels (assuming the image frame subsequence has 48 frames). (The last sentence appears to be incomplete and possibly refers to a different image.) 48 17 64 A heatmap of 64 is used as input to the PoseC3D algorithm, which can extract motion features from each image frame subsequence through model inference. The input dimensions of the PoseC3D algorithm can be represented as (B, T, N, H, W), where B is the batch size, T is the time sequence length or number of time steps, N is the number of keypoints (Nodes), and H and W are the height and width of the heatmap, respectively. In this case, B=1, T=48, N=17, H=64, and W=64.

[0054] Furthermore, when using the BlockGCN algorithm as an action feature extraction algorithm, the input dimension of the BlockGCN algorithm can be represented as (B, T, N, C), where C is the node feature dimension (Channels), used to represent keypoint coordinates as two-dimensional coordinates (C=2) or three-dimensional coordinates (C=3). The BlockGCN algorithm is an efficient graph convolution improvement model for skeletal action recognition. Its core is to combine block convolution (BlockGC) and topological coding to solve the problems of topological forgetting, weak multi-relation modeling, and low computational efficiency of traditional GCN on skeletal data. It achieves fewer parameters and higher accuracy on skeletal datasets such as NTU RGB+D.

[0055] With the current accuracy of skeletal keypoint detection, relying solely on skeletal heatmaps (i.e., 1) 48 17 64 The 64-dimensional heatmap is sufficient to meet the core requirements for feature extraction from image frame subsequences. To further improve the feature extraction accuracy of the PoseC3D algorithm, in addition to the core multi-dimensional skeletal keypoint heatmap input, RGB images can be introduced as auxiliary input. This integrates visual information such as scene texture, product location, and human pose details from the RGB images and the original image frames, compensating for the feature deficiencies of a single skeletal modality in low-occlusion, high-resolution scenes, and helping the action feature extraction algorithm more accurately distinguish similar behaviors or actions. Specifically, the heatmap matrix corresponding to each image frame subsequence can be input into the PoseC3D algorithm; or the heatmap matrix and the RGB image sequence corresponding to each image frame subsequence can be input together into the PoseC3D algorithm. In this case, image frames in surveillance video streams are typically compressed into YUV or encoded for transmission, while RGB images can be obtained by decoding and color space conversion of all image frames in the image frame subsequence.

[0056] Step S140: For the action feature representation corresponding to each image frame sub-sequence, determine the semantic similarity between the action feature representation and each standard feature representation in the pre-stored standard feature representation-to-token mapping relationship, and select the token corresponding to the action feature representation from the standard feature representation-to-token mapping relationship based on the semantic similarity. By combining the tokens corresponding to all image frame sub-sequences in a temporal sequence, the token sequence corresponding to the monitoring video stream can be obtained.

[0057] More specifically, the specific execution process of step S140 is as follows: For the action feature representation corresponding to each image frame sub-sequence, determine the Euclidean distance between the action feature representation and each standard feature representation in the pre-stored standard feature representation-lexical mapping relationship, and determine the semantic similarity between the action feature representation and each standard feature representation based on the determined Euclidean distance; for each action feature representation, take the standard feature representation with the highest semantic similarity as the standard feature representation corresponding to the action feature representation, and take the corresponding lexical of the standard feature representation with the highest semantic similarity in the mapping relationship as the lexical corresponding to the action feature representation. This application does not specifically limit the method of determining semantic similarity based on Euclidean distance; the method can be designed following the principle that Euclidean distance and semantic similarity are proportional.

[0058] In some embodiments of this invention, the standard feature representation-lexical mapping relationship is obtained by combining the behavioral characteristics in the shelf scenario and employing the BPE clustering strategy. The core logic of the BPE clustering strategy is to transform the high-dimensional continuous feature representation into a discrete "meta-action sub-word" with semantic relevance (the meta-action sub-word is the standard feature representation). Then, a standard feature representation-lexical mapping dictionary can be constructed through encoding to ensure coverage of all key actions without redundancy. That is, this application designs a standard feature representation corresponding to a meta-action of theft behavior, and the meta-actions corresponding to all standard feature representations in the standard feature representation-lexical mapping relationship include at least all meta-actions of theft behavior (such as "reaching the product", "taking the product", "returning the product", "hiding the product", and "putting the product into the shopping cart"). Specifically, the BPE clustering strategy uses the action feature representation extracted from the composite surveillance video stream as a clustering unit, and clusters these clustering units based on semantic similarity to achieve the discretization and semantic structuring of continuous features; and the discrete standard feature representation can be transformed into a discrete, computable lexical form, thereby realizing the behavior recognition of the temporal feature sequence of actions with continuous and fuzzy semantics. The standard feature representation-lexical mapping relationship can be obtained in the following way: Step S01: Obtain composite surveillance video streams in the shelf scene. This application requires that the composite surveillance video streams obtained contain all meta-actions of theft. Therefore, a large number of video frame fragments from multiple shelf areas, different time periods, and different personnel targets in the shelf scene can be collected (ensuring coverage of core theft meta-actions such as "taking, putting back, touching, hiding, and putting into the shopping cart") to form a video stream set (i.e., composite surveillance video stream) including multiple shelf areas, multiple personnel targets, and / or multiple identifiable theft meta-actions.

[0059] Each video stream in the video stream set can include the complete process of an interaction between a specific person and a product at a specific shelf location. This application does not specifically limit the types of identifiable theft meta-actions, such as "touching the product," "taking the product," "returning the product," "hiding the product," and "adding the product to the shopping cart," etc., which can be designed according to the shelf scenario.

[0060] Step S02: Using keypoint detection, segmentation, and motion feature extraction algorithms, obtain the action feature representations of personnel targets corresponding to each image frame subsequence in the composite surveillance video stream. The algorithms and execution process in this step are the same as in steps S120 and S130.

[0061] Step S03: Using a clustering algorithm, the action feature representations corresponding to all image frame subsequences in the composite surveillance video stream are used as clustering units for clustering. Semantically similar action feature representations are clustered into one class, thus obtaining multiple clusters. This application does not specifically limit the type of clustering algorithm. For example, the HDBSCAN algorithm can be used, which does not require a preset number of clusters and can automatically divide clusters through a density threshold. At the same time, it can effectively filter out abnormal features (such as low-quality features caused by occlusion).

[0062] More specifically, in the clustering process, the Euclidean distance in the feature space can be used as a semantic similarity metric. That is, the closer the Euclidean distance between action feature representations, the more similar the corresponding meta-action semantics (e.g., the feature vectors of "take the upper layer of goods with the right hand" and "take the middle layer of goods with the right hand" may be clustered into one class).

[0063] Step S04: Take the cluster center of each cluster as a standard feature representation, and construct a standard feature representation-term mapping relationship (i.e., a standard feature representation-term mapping dictionary in the shelf scenario) by labeling each standard feature representation with terms (e.g., the cluster ID of the standard feature representation can be directly used as a term). The standard feature representation can be a clustering unit, or it can be obtained based on the clustering unit (rather than the clustering unit itself).

[0064] After clustering, the center vector of each cluster is designated as the standard feature representation, which is essentially the "semantic prototype feature" of all similar meta-actions within that cluster. For example, the cluster center that gathers a large number of action feature representations of "taking goods" is the standard feature representation of the meta-action "taking"; the cluster center that gathers action feature representations of "putting back goods" corresponds to the meta-action "putting back". Through this clustering process, the originally scattered and continuous high-dimensional feature vectors can be mapped to a discrete standard feature representation space. The action feature representation of each new image frame sub-sequence can be matched with the best-fitting standard feature representation by calculating the distance to each cluster center. This ultimately achieves the transformation from "image frame sub-sequence → action feature representation → standard feature representation corresponding to a specific meta-action", providing core support for the behavioral semantic interpretation in the subsequent step S150.

[0065] Step S150: According to the temporal sequence of each image frame subsequence in the surveillance video stream, obtain the word sequence corresponding to the surveillance video stream, input the word sequence into the pre-trained semantic classification model, and output the identification result of whether the surveillance video stream contains theft behavior.

[0066] Considering that a person's actions or events may involve multiple behaviors during the entire time they are in front of the shelf, this application performs semantic understanding on long video streams based on segmentation to identify whether theft has occurred. The purpose of semantic understanding is to reflect the action logic of the person's target. That is, the core of the semantic understanding of the interpretable word sequence in step S150 of this application is to enable the semantic classification model to "understand" the standard feature representation logic behind the word sequence (such as "reaching the product sub-word → taking the product sub-word → hiding the product sub-word → leaving the sub-word" = theft). Since the word sequence originates from the standard feature representation determined by BPE clustering, the word corresponding to each image frame sub-sequence can carry accurate action semantic information. By analyzing the word sequence of the person's target in the shelf scene (the core is the standard feature representation generated based on the BPE clustering strategy), the interaction relationship between the person's target and different product targets can be accurately restored. This allows the semantic classification model to accurately capture the semantic differences between different action chains such as "taking → putting back" and "taking → hiding", realizing the transformation from "identifying action features" to "understanding action logic".

[0067] In some embodiments of the present invention, the pre-trained semantic classification model can be constructed based on a Transformer neural network. After the temporal token sequence (i.e., word sequence) obtained by converting the surveillance video stream in step S140 is input into the semantic classification model, the Transformer can capture the temporal dependencies and semantic associations of the temporal token sequence through an attention mechanism (such as a multi-head attention mechanism), and automatically focus on key words (such as increasing the weight of the word "hidden goods" by more than 30%), transforming discrete words into distinctions between "normal behavior" and "theft behavior", thereby accurately distinguishing the semantic differences between similar action chains such as "taking → putting back" and "taking → hiding", and achieving a deep understanding of behavioral logic. It should be noted that after the labeled standard feature representation is converted into words and input into the Transformer network in step S140, the Embedding layer of the Transformer network will use words as discrete indices, assign a new feature vector value to each word, and dynamically adjust the vector representation of each word through subsequent multi-layer self-attention mechanisms. If continuous standard feature representations are directly input into the Transformer network, it will destroy the embedding layer, disrupt sequence modeling, and cause the model training to crash.

[0068] The aforementioned method of constructing a semantic classification model based on Transformer is merely an example. This invention does not specifically limit the type of semantic classification model; any model capable of semantically understanding word sequences is acceptable. A pre-trained semantic classification model can be obtained as follows: acquire the training word sequences corresponding to training samples (including training surveillance video streams and theft behavior labels); input the training word sequences into the semantic classification model to be trained, and iteratively adjust the model parameters based on the model output and theft behavior labels to obtain the pre-trained semantic classification model.

[0069] As an example, semantic classification models can improve the accuracy of identifying theft in shelf scenarios by accurately interpreting word sequences (which have temporal order) and circumventing the time length limitations of surveillance video streams. However, before converting the action feature representation sequence corresponding to the surveillance video stream into a word sequence to input into the pre-trained semantic classification model, the word sequence needs to be preprocessed to ensure that the word sequence length of the input model is uniform. The preprocessing method for the word sequence can be as follows: if the length of the converted word sequence is less than a set sequence length threshold, pad it with zeros; otherwise, truncate redundant words without semantic meaning (such as words that clearly represent meaningless behavior), so that the processed sequence reaches the set sequence length threshold. The above method for ensuring uniform sequence length is only an example and can be designed independently.

[0070] This invention uses the complete video stream as the target for behavior recognition and performs semantic understanding on various behaviors or events that may occur in the video stream to draw holistic conclusions about the surveillance video stream. This application aims to capture the sequential logic of actions and behavioral intent. For surveillance video streams of a certain length containing multiple complex behaviors and events, it can clearly distinguish between normal behaviors such as "touching the product," "taking and putting it back," "taking and taking it away," and "taking it down and putting it in the shopping cart," and theft behaviors such as "taking and hiding" and "stealing the product while avoiding surveillance."

[0071] Furthermore, in this invention, the entire process of frame segmentation, action feature extraction, selection of standard feature representations, and word sequence analysis corresponds one-to-one with the time frames of the surveillance video stream. That is, each image frame subsequence corresponds to a fixed frame range, and all image frame subsequences, action feature representations, standard feature representations, and words can be directly traced back to the time nodes of the surveillance video stream. Based on this timestamp association mechanism, complete key video segments containing theft can be extracted, forming visual corroborating information labeled with timestamps and standard feature representations.

[0072] Compared to traditional methods that detect keyframes in isolation using a single model or lose action details through frame sampling, the theft behavior recognition method proposed in this application, which combines "meta-action segmentation - action feature extraction - standard feature representation calibration - lexical encoding - behavioral semantic understanding," can achieve the following beneficial effects through computer vision analysis and scenario-based optimization design, and also has practical application value: ① Complete behavior chain analysis can improve the accuracy of theft behavior identification and reduce the false positive and false negative rates: This invention is not limited to making judgments based on actions identified in a single image frame, but rather performs fine-grained meta-action decomposition of the complete behavior of personnel in front of the shelf. It extracts temporal features representing human posture behavior from the surveillance video stream (i.e., extracts features of actions such as no interaction, reaching, taking down, putting back, and putting into (shopping cart) and arranges them in chronological order), and introduces a temporal action modeling step (converting the action feature representation sequence corresponding to the surveillance video stream into a word sequence). Through deep temporal semantic analysis, it mines the internal logic of the action chain, thereby accurately distinguishing the action intent and restoring the complete action chain of "reaching-taking-putting back / hiding-leaving", significantly reducing the misjudgment of similar behaviors caused by the lack of key information.

[0073] ② Video streams are not limited by time: The analysis of video streams supports full analysis of video streams without time length limitations. Regardless of video duration or behavior overlay scenarios, accurate behavior recognition and early warning can be achieved through semantic understanding of the complete monitoring video stream.

[0074] ③ Precise time frame tracing and visual evidence: The timestamp association mechanism can not only provide precise guidance for security personnel's immediate response, but also serve as an objective basis for subsequent manual review and incident tracing, effectively solving the pain points of traditional anti-theft systems such as "no basis for alarms and difficulty in accurate tracing", and improving the credibility and efficiency of theft behavior judgment.

[0075] ④ Real-time monitoring and accurate alerts: By fusing human posture sequences (such as skeletal motion sequences), product trajectory sequences, and scene context features, the interactive relationship between personnel and products can be established, accurately capturing the complete time information of personnel's "picking up-putting back-leaving" process. Through analysis of complete monitoring video streams during customer-shelf product interactions, customer behavioral intentions can be understood, reducing false alarms for normal customer shopping and thus improving the operational capabilities and user experience of the shelf environment. Furthermore, the monitoring video stream to be identified can be acquired in real time when personnel leave the shelf and / or when the product position stops changing, thereby reducing motion recognition latency.

[0076] ⑤ Reduce anti-theft operation costs and adapt to large-scale deployment: Compared with common shelf theft detection methods, this invention does not require an excessive number of cameras. It can reuse existing cameras and analyze the video streams captured by the cameras, which can greatly reduce hardware modification costs. Furthermore, this application can replace manual monitoring by intelligently identifying theft behavior through computer vision solutions, thereby reducing manual verification costs.

[0077] ⑥ Immediate handling: This application can automatically simplify the long-term complex behavior of personnel in front of the shelf into a word sequence by acquiring the monitoring video stream in real time, segmenting the video stream, extracting action feature sequences, converting them into word sequences, and semantic understanding, thereby improving the timeliness of anti-theft.

[0078] In the research, development, training, and testing process of this technical solution, which involves processing personal images from public places, the following strict safeguards are implemented for such images: ① The primary and sole purpose of all images from public places is necessary for maintaining public safety (specifically, identifying theft and responding to public safety emergencies); ② Clear and prominent signage has been installed in the image area to clearly inform the public of the collection area, purpose, information type, and responsible agency, ensuring the public's right to know; ③ All collected information is used solely for the aforementioned public safety purposes and will not be used for any commercial marketing, personalized recommendations, or other purposes unrelated to maintaining public safety; ④ Information is stored in a dedicated system or controlled experimental environment with high-level security protection, implementing strict access control and operational auditing; ⑤ In research scenarios involving non-real-time security operations such as model training and algorithm verification of this technical solution, the original data is thoroughly anonymized or de-identified to ensure that it cannot be identified or associated with any specific natural person.

[0079] Corresponding to the above method, the present invention also provides a theft behavior recognition system in a shelf setting. The system includes a computer device, which includes a processor and a memory. The memory stores computer programs / instructions, and the processor is used to execute the computer programs / instructions stored in the memory. When the computer programs / instructions are executed by the processor, the system implements the steps of the method described above.

[0080] Those skilled in the art will understand that the exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention. When implemented in hardware, it can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this invention are programs or code segments used to perform the desired tasks. The programs or code segments can be stored in a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried in a carrier wave.

[0081] It should be clarified that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of the present invention.

[0082] In this invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or in place of features of other embodiments.

[0083] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations of the embodiments of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for identifying theft behavior in a shelf setting, characterized in that, The method includes the following steps: Acquire a surveillance video stream of the shelf scene; wherein the surveillance video stream includes multiple image frames containing human images; Human key point information of each image frame in the monitoring video stream is obtained by a key point detection algorithm, and the monitoring video stream is segmented by a segmentation algorithm to obtain at least one image frame subsequence; wherein, there are no overlapping image frames in any two image frame subsequences. Using an action feature extraction algorithm, the action feature representation corresponding to each image frame subsequence is obtained based on the human key point information of the image frames contained in each image frame subsequence; For each image frame subsequence, the semantic similarity between the action feature representation and each standard feature representation in the pre-stored standard feature representation-lexical mapping relationship is determined, and the lexical corresponding to the action feature representation is selected from the mapping relationship based on the semantic similarity; wherein, one standard feature representation corresponds to one meta-action of theft, and the meta-actions corresponding to all standard feature representations in the mapping relationship include at least all meta-actions of theft. Based on the temporal sequence of each image frame subsequence in the surveillance video stream, the corresponding word sequence of the surveillance video stream is obtained, and the word sequence is input into a pre-trained semantic classification model to output the identification result of whether the surveillance video stream contains theft behavior.

2. The method according to claim 1, characterized in that, The step of segmenting the surveillance video stream using a segmentation algorithm to obtain at least one image frame subsequence includes: The surveillance video stream is uniformly segmented, or the surveillance video stream is non-uniformly segmented based on the human body key point information to obtain at least one image frame subsequence; The step of unevenly segmenting the surveillance video stream based on the human body key point information includes: The motion displacement of each image frame is determined based on information from specific human body key points in each image frame; wherein, the specific human body key points include elbow key points and wrist key points. An action trajectory waveform is formed based on the motion displacement of all image frames in the monitoring video stream, and the monitoring video stream is segmented based on the extreme points in the action trajectory waveform.

3. The method according to claim 2, characterized in that, After segmenting the surveillance video stream based on extreme points, the method further includes: Determine the number of frames in each initial sequence obtained by segmenting the surveillance video stream based on extreme points; For each initial sequence, if the number of frames in the initial sequence does not meet the set frame count condition, the initial sequence is subjected to frame extraction, resampling, and merging processing based on a preset image sequence processing strategy. The image sequence processing strategy includes: If the number of frames in the initial sequence is less than the first set frame number threshold, then the initial sequence and the initial sequence following the initial sequence in the timing sequence are merged. If the number of frames in the initial sequence is between the first set frame number threshold and the second set frame number threshold, then the initial sequence is resampled. If the number of frames in the initial sequence is greater than the second set frame number threshold, then the initial sequence is subjected to frame extraction.

4. The method according to claim 1, characterized in that, The motion feature extraction algorithm is the PoseC3D algorithm, and the human body key point information includes key point coordinates and key point confidence. The action feature extraction algorithm, based on the human key point information of the image frames contained in each image frame sub-sequence, obtains the action feature representation corresponding to each image frame sub-sequence, including: For each image frame in each image frame subsequence, the information of each human key point is mapped to a heat map according to the confidence of each key point in the image frame. The heat map obtained by stacking all human key points in the image frame is used to form the heat matrix corresponding to the image frame. For each image frame subsequence, the heat matrix corresponding to each image frame is stacked according to the temporal order of each image frame in the image frame subsequence to obtain the heat matrix corresponding to the image frame subsequence; The PoseC3D algorithm inputs the heatmap matrix corresponding to each image frame subsequence and outputs the motion feature representation corresponding to each image frame subsequence.

5. The method according to claim 4, characterized in that, The PoseC3D algorithm, which inputs the heatmap matrix corresponding to each image frame sub-sequence, includes: Input the heatmap corresponding to each image frame subsequence into the PoseC3D algorithm; or input the heatmap corresponding to each image frame subsequence and the RGB image sequence corresponding to each image frame subsequence into the PoseC3D algorithm.

6. The method according to claim 1, characterized in that, The step of determining the semantic similarity between the action feature representation and each standard feature representation in the pre-stored standard feature representation-lexical mapping relationship, and selecting the lexical corresponding to the action feature representation from the mapping relationship based on the semantic similarity, includes: Determine the Euclidean distance between the action feature representation and each standard feature representation in the pre-stored standard feature representation-lexical mapping relationship, and determine the semantic similarity between the action feature representation and each standard feature representation based on the Euclidean distance; Select the lexical corresponding to the standard feature representation with the highest semantic similarity from the pre-stored standard feature representation-lexical mapping relationship, and use it as the lexical corresponding to the action feature representation.

7. The method according to claim 1, characterized in that, The standard feature representation-lexical mapping relationship is obtained through the following method: Acquire composite surveillance video streams in a shelf setting; wherein, the composite surveillance video stream is a set of surveillance video streams including multiple shelf areas, multiple personnel targets, and / or multiple identifiable theft actions; The action feature representations corresponding to each image frame subsequence in the composite surveillance video stream are obtained by using key point detection algorithm, segmentation algorithm and action feature extraction algorithm; Using clustering algorithms, the action feature representations corresponding to all image frame subsequences in a composite surveillance video stream are clustered to obtain multiple clusters. The cluster centers of each cluster are used as standard feature representations, and the standard feature representation-lexical mapping relationship is constructed by labeling the standard feature representations with lexical units.

8. The method according to claim 1, characterized in that, The acquisition of the monitoring video stream in the shelf scenario includes: For shelf scenarios, acquire raw monitoring video streams containing multiple image frames; The location of goods and / or people in each image frame of the original surveillance video stream is determined by the target detection algorithm. The starting and ending image frames of the surveillance video stream are determined based on the target location of the goods and / or the target location of the personnel, thereby extracting the surveillance video stream from the original surveillance video stream.

9. The method according to claim 1, characterized in that, The pre-trained semantic classification model is built based on Transformer; the pre-trained semantic classification model is obtained in the following way: Obtain the training word sequence corresponding to the training sample; The training word sequence is input into the semantic classification model to be trained, and the model parameters are iteratively adjusted based on the model output to obtain the pre-trained semantic classification model.

10. A theft detection system in a shelf setting, comprising a processor, a memory, and a computer program / instructions stored in the memory, characterized in that, The processor is configured to execute the computer program / instructions, and when the computer program / instructions are executed, the system implements the steps of the method as described in any one of claims 1 to 9.

Images