Method and apparatus for anomaly detection in a self-checkout retail environment

Abstract

A system for anomaly detection in a self-service checkout environment includes a processing unit, the processing unit being configured to: extract a set of features from transaction data received from a self-service checkout terminal; characterize an activity based on the set of features; define a plurality of activity intervals for each characterized activity; determine a meta-feature vector for each defined activity interval in the plurality of activity intervals; compare each meta-feature vector with a set of predefined vectors; and detect anomalies based on the comparison.

Description

Technical Field

[0001] The present invention relates to a method and apparatus for anomaly detection in a self-checkout retail environment. Background Art

[0002] Self-checkout and automated processes are becoming increasingly mainstream in retail environments. However, this leads to a drawback: retail environments based on self-checkout (SCO) often result in lower customer traffic for a variety of reasons. Consumers or customers are usually not trained on self-checkout mechanisms and in many cases feel uncomfortable with the technology. The customer's lack of knowledge and experience slows down the process of checking out and paying for items. For example, a customer may perform the scanning process incorrectly and must wait for a retail employee to reset the machine or provide authorization for the transaction, or in fact the customer may simply get hung up on scanning the relevant barcode. Obviously, deliberate attempts to avoid transactions can also occur. Time lost waiting for invalid transactions, weighing scales that need to be calibrated, and additional non-ideal scenarios lead to interruptions. These non-ideal scenarios include non-scanning, overcharges, invalid transactions, etc.

[0003] It is understandable that disruptions in customer flow, such as those described above, can affect profitability and turnover; for example, considering peak activity periods. Flow disruptions delay customer throughput, can cause backlogs, and / or can annoy customers who would otherwise make some impulse purchases. This in turn affects turnover and reduces the efficiency of the self-checkout system. Retailers have previously attempted to control this problem through manual monitoring of the SCO by attendants / operators. An attendant's job typically has two parts: a monitoring component and a decision / action component. The monitoring component is when the attendant is required to observe any irregularities and blockages at the SCO, while the action component is when the attendant attempts to correct the problem when an irregularity is observed. Obviously, this also leads to irregularities in flow. Attendants are often inclined to reset the machine and ignore the fact that the product has not been scanned. This is a loss for the retail owner and can be abused. Irregularity is defined as any activity that does not follow the expected pattern of the shopping process. The shopping process consists of a series of actions: picking an object from a certain area, placing an object in a certain area, passing an object through a scanner, pressing a button or a certain touch screen area, bagging a product, passing with a non-empty shopping cart, etc.

[0004] Therefore, there is a need to implement automated methods and devices to optimize passenger flow. Summary of the invention

[0005] It should be understood that all combinations of the foregoing concepts and the additional concepts discussed in more detail below (assuming that these concepts are not mutually contradictory) are considered to be part of the inventive subject matter disclosed herein. In particular, all combinations of the claimed subject matter appearing at the end of this disclosure are considered to be part of the inventive subject matter disclosed herein. It should also be understood that those terms explicitly adopted herein that may also appear in any disclosure incorporated by reference should have meanings consistent with the specific concepts disclosed herein.

[0006] In one aspect, the present invention provides a system for anomaly detection in a self-service checkout environment, comprising a processing unit for: extracting a set of features from transaction data received from a self-service checkout terminal; characterizing activities based on the set of features; defining a plurality of activity intervals for each characterized activity; determining a meta-feature vector for each defined activity interval in the plurality of activity intervals; comparing each meta-feature vector with a set of predefined vectors; and detecting anomalies based on the comparison.

[0007] Characterizing the activity may include extracting a set of features from the received data, converting the set of features into a corresponding set of feature activation values ​​and determining the evolution of the set of activation values ​​within a time interval, and defining the activity according to the following equation μ :

[0008] {f i (n)|i∈featureSet μ ,μ∈[1,K],n∈[N1,N2],N2=N1+N a},

[0009] Where N a is the number of frames in which the activity is considered to be detected.

[0010] f i (n) is the value of the extracted feature group,

[0011] featureSet μ Describes the Activity μ A set of characteristics,

[0012] [N1, N2] represents a group of consecutive frames extracted within the time interval between the time of the initial frame N1 and the time of the last frame N2.

[0013] The processor may be further configured to estimate the number N of frames defining the activity a The estimation consists in defining the activation function A(f i (n) f R i ) and divide the time domain into active intervals based on this function, where A(fi (n) f R i ) is characterized as

[0014] Where R i Represents the user for each feature f i A set of rules defined; when R i For a given frame n, when f i (n)To be active.

[0015] For all K activities, each active interval can be defined as

[0016]

[0017] The processor may be further configured to determine inactive intervals between adjacent active intervals.

[0018] The processor may be further configured to include comparing the inactive interval to a threshold and combining the adjacent active intervals and the inactive intervals into a merged active interval.

[0019] Determining the meta-feature vector may include for each feature f in the active interval i A number of statistical measures are calculated and the features are combined to form a meta-feature vector for each active interval.

[0020] Statistical measures may include statistical moments and / or ordered statistics.

[0021] The processor may be configured to perform a first comparison between the determined meta-feature vector and the first classification system model and to create an alert based on the comparison.

[0022] The processor may be configured to perform a second classification comparison between the determined meta-feature vector and the second system model.

[0023] The processor may be configured to compare performance of the first system model and the second system model based on the first and second comparisons, and replace the first system model with the second system model if the performance of the second system model is better than the first system model.

[0024] The system may further include a memory for storing the first system model as a backup model.

[0025] The processor may be configured to receive feedback to the alarm and update the second system model based on the feedback.

[0026] The processor may be configured to compare the performance of the first system model with the performance of the backup model and replace the first system model with the backup model if the backup model outperforms the first system model.

[0027] The system may further include a plurality of reporting devices for reporting the alarm. The reporting device may be selected from a list including a mobile computing device, a mobile communication device, a smart watch, smart glasses, a personal computer, a networked server, a visual display unit, and an audio output device. The system may further include a POS device.

[0028] The system may further include a plurality of sensors for sensing transaction data from the self-service checkout terminal.

[0029] Another embodiment of the present invention includes a method for anomaly detection in a self-service checkout environment, comprising: extracting a set of features from transaction data received from a self-service checkout terminal; characterizing an activity based on the set of features; defining a plurality of activity intervals for each characterized activity; determining a meta-feature vector for each defined activity interval in the plurality of activity intervals; comparing each meta-feature vector with a set of predefined vectors; and detecting anomalies based on the comparison.

[0030] The method may further include: characterizing the activity by extracting a set of features from the received data, converting them into a corresponding set of feature activation values ​​and determining the evolution of the set of these values ​​over a time interval, and then defining the activity according to the following set μ :

[0031] {f i (n)|i∈featureSet μ ,μ∈[1,K],n∈[N1,N2],N2=N1+N a}

[0032] Where K represents the total number of activities recorded in the system knowledge base, featureSet μ Describes the Activity μ A set of features, N a is the number of frames where activity is considered detected, f i (n) is the value of the extracted feature group, and [N1, N2] represents a group of consecutive frames extracted within the time interval between the time of the initial frame N1 and the time of the last frame N2.

[0033] The method may further comprise estimating the number of frames defining said activity, N a is estimated, and wherein the estimation includes defining an activation function A(f i (n) f R i) and divide the time domain into active intervals based on the function, and where A(f i (n) f R i ) is characterized as

[0034] Where R i Represents the user for each feature f i A set of rules defined; when R i When true, f i (n)To be active.

[0035] For all K activities, each active interval can be defined as

[0036]

[0037] The method may further include determining inactive intervals between adjacent active intervals.

[0038] The method may further include comparing the inactive interval to a threshold, and combining the adjacent active intervals and the inactive intervals into a merged active interval.

[0039] The method may further include: wherein determining the meta-feature vector comprises for each feature f in the active interval i A number of statistical measures are calculated and the features are combined to form a meta-feature vector for each active interval.

[0040] Statistical measures may include statistical moments and / or ordered statistics.

[0041] The method may include performing a first comparison between the determined meta-feature vector and the first classification system model and creating an alert based on the comparison.

[0042] The method may include performing a second classification comparison between the determined meta-feature vector and the second system model.

[0043] The method may include comparing performance of the first system model and the second system model based on the first and second comparisons, and replacing the first system model with the second system model if the performance of the second system model is better than the first system model.

[0044] The method may further include storing the first system model as a backup model.

[0045] The method may further include receiving feedback to the alarm, thereby updating the second system model based on the feedback.

[0046] The method also includes comparing the performance of the first system model to the performance of the backup model and replacing the first system model with the backup model if the backup model outperforms the first system model.

[0047] A computer program is also provided, which comprises program instructions for causing the computer program to implement the above method, and the program instructions may be embodied on a recording medium, a carrier signal or a read-only memory. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Embodiments of the present invention will now be described, by way of example only, with reference to the accompanying drawings, in which:

[0049] Figure 1 A self-checkout environment is depicted according to an embodiment of the present invention.

[0050] Figure 2 is a flow chart depicting an example of a processing unit according to an embodiment of the present invention.

[0051] Figure 3 is a flow chart depicting an example of an attendant's decision-making process in response to an alert.

[0052] Figure 4A and Figure 4B A schematic diagram of an active interval according to the present invention is provided.

[0053] Figure 5 is the meta-feature vector according to the present invention.

[0054] Figure 6 A classification and reporting model according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0055] According to an embodiment of the present invention, there is provided a self-service checkout environment 100, a processing unit 102, and a reporting mechanism 103. The reporting mechanism includes a plurality of reporting devices.

[0056] The self-checkout environment includes a point of sale device or self-checkout machine and a sensor network 101. The sensor network includes a plurality of sensors. The sensors may include camera(s), audio sensor(s), 3D scanner(s), etc. The sensors may include any sensor capable of extracting information related to a retail transaction, such as a barcode, QR code, audio, video or still image associated with the transaction, etc. During a transaction, transaction data is extracted at the sensor network (101) and provided to a processing unit (102). The transaction data includes transaction data such as barcodes, QR codes, audio recorded during the transaction, transaction video and still images. Transaction data is data related to the transaction.

[0057] The processing unit (102) can be located in the retail environment or can be remote from the retail environment, such as at a central hub. The central hub can be networked to multiple retail environments. For example, the central hub can be responsible for processing data from multiple retail sites, chain retailers, etc. The processing unit processes data from the sensor network. In response to the processing, an alarm or information message can be reported to multiple reporting devices 103. These reporting devices can include online reporting devices, such as computing terminals, such as machines, remote units, personal computing devices, wireless communication devices, smart watches, smart glasses, pagers, etc. The reporting devices can also include many offline reporting devices, including database servers, cloud servers, etc. Both online and offline reporting devices are connected through a secure network. These information messages can be provided to one or more of these devices, such as to operators 104 and / or supervisors or managers. In addition, the report can also be addressed to the customer, such as alerting the customer of the detected anomaly. It should be understood that instructions can be provided to the customer to correct the anomaly. The reporting message can be a standard phrase pre-recorded using an audio output and / or a written message on a SCO display, for example. This type of customer reporting will minimize the need for operators and thereby maintain traffic. However, the usefulness is limited in fraud situations. As an alternative, for operator alerts, when an event is detected and operator intervention is required, a message with the above information can be issued.

[0058] Figure 2 The processing unit of includes an analysis unit 201 and a decision unit 204. Although shown as separate units, it should be understood that these units can be combined into a single unit. The sensor network extracts data that can be used to characterize retail transactions. This extracted data is transaction data and is collected by sensors installed in the point of sale environment 101. The transaction data includes data such as barcodes, QR codes, audio recorded during transactions, transaction videos and still images. By analyzing the data, the steps of the retail process can be identified. The steps include, for example, picking products, scanning products, or bagging products. The data is passed to the analysis unit 201 that processes the data. The processing of the data is used to extract meta-features from the data. The information is then passed to the decision unit. The decision unit makes a decision on whether there is an irregularity in the data. Once a decision is made on the irregularity, an alarm or information message is sent to the relevant reporting device. The relevant reporting device can be pre-selected and individually configured for each retail environment. The information message can be just an alarm about the irregularity. In addition, information about the type of irregularity can be provided to enable easy detection.

[0059] As described above, information messages are provided via a network of online and offline devices. It should be understood that communication with online and offline devices can optionally be bidirectional with one or all devices. An operator can use a bidirectional device to provide feedback on received information messages. Feedback can include confirmation / rejection of an alarm. Feedback is sent to a decision unit (204) by a reporting device. Feedback is used to adjust the algorithm. This adaptive algorithm improves accuracy, robustness to false positives, and detection speed. For example, feedback occurs when an alarm is issued but the waiter's assessment indicates a false positive. Relevant system data is input so that the adaptive components of the model can be updated.

[0060] It should also be understood that the attendant or operator can provide spontaneous feedback. For example, in the case where no alarm or message is provided, but the operator detects an irregularity, feedback can be provided. This information can be used to create new alarm or problem instances. Feedback on the attendant's intervention in the retail process (including time spent, actions taken, and impact on the process) can also be provided.

[0061] For example, in Figure 3 At step 301, an attendant monitoring a self-checkout will wait for an alert. The attendant will receive the alert and optionally relevant data about the detected irregularity 302. Based on the alert and information, the attendant evaluates the alert and makes a decision as to whether an event exists 306. If an event is detected, the attendant takes appropriate action to the customer 308 and provides relevant feedback. If there is no event, the attendant provides feedback 310 with an objective assessment based on the reasoning behind his decision.

[0062] Another example, this time involving reporting to customers, involves providing some pre-recorded standard phrases using audio output and / or written messages on the SCO display. Such messages guide the customer through the retail process (e.g., "Please rescan the product!"). This type of reporting will minimize the need for operators and thereby maintain the process.

[0063] about Figure 3 The described method aims to replace and assist the decision-making and action parts of operators and salespeople.

[0064] The extraction of meta-features provides a basis for determining whether there are suspicious events or irregularities in the self-checkout process. The feedback as described above is used to build and adjust the decision model to include new instances and cases of irregularities and can be used to improve accuracy.

[0065] More details about the irregularity or anomaly detection mechanism are described below.

[0066] As about Figure 1As described, information or transaction data is obtained from multiple sensors. The information may include, for example, video data that provides image streams associated with retail and self-service checkout processes. The method described below describes video data, but any data may be used. For each video frame (n), the output of one or more sensors of the sensor network is sampled to extract a set of features from the transaction data at a discrete time n. It should be understood that the sensor network monitors multiple activities simultaneously. Features may include, for example, the percentage of foreground pixels in a video frame, the number of motion vectors in a specific area of ​​a video frame, the orientation, or the number of key points present in an area of ​​interest in a video frame. By applying statistical functions to the features, meta-features are obtained based on feature changes within a time interval, such as statistical moments (mean, variance, skew, kurtosis, etc.) or ordered statistics (minimum, maximum, median, quartiles, trimmed mean, median absolute deviation, etc.). In the case of Y features, a number of N×Y statistical measures will be calculated and encoded into the meta-feature vector. Each Activity μ By featureSet μ The sensor network is characterized by a predefined subset of features and a set of meta-feature values ​​corresponding to this subset. Thus, when a particular activity is occurring or ongoing, the corresponding feature subset will be detected as active by the sensor network based on the non-zero value of the activation function. Different activities will correspond to different feature subsets. The actual feature subset selected depends on the detected activity. Thus, an activity is characterized based on its set of features. These features are represented by information relevant to solving a specific task related to the application. They can be selected manually or automatically. Automatic selection can be achieved using neural networks or other machine network techniques. Thus, a feature is represented by the value of a variable or a set of variables. In our application, features can be extracted from videos (e.g., skin detection, motion detection, motion pattern detection, foreground / background detection, body part detection, etc.), and others can be generated by sensors (e.g., detection of sounds associated with scanning, static information, etc.). These features can have different levels of complexity from the perspective of computational workload. Low-level features are represented by features extracted directly from sensors or obtained by simple computer vision algorithms (usually these simple features are obtained at the pixel level). Examples of low-level features: detected edges, detected lines, detected skin pixels, dominant color, presence of a specific sound, timestamp of a scan, signal extracted from a scanning machine, etc.) Higher-level features can be obtained by combining low-level features applying specific algorithms.

[0067] Examples of features extracted from a specific ROI (region of interest):

[0068] -Presence / absence of hand

[0069] -Percentage of foreground pixels

[0070] - The percentage of motion vectors whose projection in a given direction exceeds a specified threshold

[0071] - The dominant motion vector direction within the specified angle range

[0072] The activities occurring in the time interval between two subsequent frames N1 and N2 (expressed as [N1, N2]) can be represented by the feature set f according to the following formula: i (n) is characterized by the evolution of:

[0073] {f i (n)|i∈featureSet μ ,μ∈[1,K],n∈[N1,N2],N2=N1+N a},

[0074] Where K represents the total number of activities recorded in the system knowledge base, featureSet μ Describes the Activity μ A set of features, N a is the number of frames where activity is considered detected, f i (n) is the value of the extracted feature set, and [N1, N2] represents a set of consecutive frames extracted in the time interval between the time of the initial frame N1 and the time of the last frame N2. The frame numbers N1 and N2 are specific to each activity. Therefore, before performing the meta-feature vector calculation, the active interval [N1, N2] should be iteratively determined for each activity stored by the model. It can be determined based on an active interval detection method. In practice, the interval depends on the specific feature subset determined by the activity.

[0075] It should be understood that the number of frames evaluated to define each feature varies, namely N a is variable, i.e. not a constant. N a Varies from person to person and is practical for a single person repeating a single activity. During a checkout of multiple products, each product scan may be different for a single person. With a constant N a The sliding window approach of will produce many errors because the window size is not optimized for transactions. Suboptimal window size will lead to activity detection errors.

[0076] To optimize the window size and thus improve the efficiency of the detection mechanism, this paper recommends the following:

[0077] (i) Estimating the optimal window size

[0078] (ii) Define the time domain features normalized with respect to the window size

[0079] Estimation N a

[0080] Consider again the feature set f extracted to characterize the activity i (n):

[0081] For each feature group f i (n) defines an activation function.

[0082] (i) Where R i Represents the user for each feature f i A set of rules defined; when R i When true, f i (n)To be active.

[0083] For example, a user-defined rule could be as follows:

[0084] (i)f i (n)≤value

[0085] (ii) Value 1≤f i (n)≤value2

[0086] (iii)∫f i (n)<0

[0087] Considering equations i to iii, “value, value1, value2” define the limit values ​​of the characteristic. Assume that characteristic f i (n) is normalized to take values ​​between 0 and 1, then a value of 0.5 will generate a rule that only if f i (n) Feature f is activated only when its value is less than half of its maximum span i (n). Formula (ii) activates the feature only when its value is between value 1 and value 2. Extreme values ​​like value 1 and value 2 are set during the design process using standard machine learning methods or through trial and error.

[0088] For each feature f i , activation function A(f i (n) f R i ) is used to divide the time domain into activation intervals (the function takes the value of 1 within an interval). For all K activities, each active interval can be defined as

[0089] (i)

[0090] As an example, Figure 4A As shown in , four features are considered, namely feature 1, feature 2, feature 3 and feature 4. Figure 4A Four characteristic traces are shown, each of which represents the function A(f i (n), R i). Four active intervals 401, 402, 403, 404 are shown. Also shown are gaps, i.e. inactive intervals between adjacent active intervals 405, 406, 407. These gaps are caused by noise in the system.

[0091] For feature 1 in the first active interval 401 , feature 1 is detected once. In the second active interval 402 , feature 2 is detected once within a shorter period of time. Feature 1 is not detected in the third active interval 403 or the fourth active interval 404 .

[0092] Feature 2 is detected once in the first active interval 401 and again in the third active interval 403. Feature 2 is not detected in the second active interval 402 or the fourth active interval 404.

[0093] Feature 3 is detected in the first active interval 401 , the second active interval 402 , and the fourth active interval 404 , whereas feature 3 is not detected in the third active interval 403 .

[0094] Considering the feature trace of feature 4, there are two active intervals. In the first active interval 401, feature 4 is detected twice. In 404, feature 4 is not detected.

[0095] The active intervals for each activity vary in length. Figure 4A . The intervals with predefined thresholds for activity detection may be predefined based on a determination of intervals typically associated with routine checkout activities. The size of the gaps may vary. For example, gap 407 is smaller than gap 405.

[0096] To eliminate small gaps or inactive intervals, a “loose union” of intervals is performed to merge two active intervals into a single interval, e.g. Figure 4B As shown, especially with regard to Figure 4B The active intervals 403 and 404 are merged into a single active interval 408 by merging the gap interval 407 in the active interval.

[0097] To determine the merging interval, the gaps between active intervals ξ are compared to a threshold τ. The optimal value of τ (threshold for the gaps) is determined using a database of extracted active intervals. The value of τ is obtained as the value that maximizes the accuracy of the subsequent classifier. This is done experimentally. Valid gaps for loose union are considered to be gaps that are smaller than a threshold (e.g. 300 milliseconds). Larger gaps cannot be included in the active intervals. The threshold can be learned (by any existing method) or can be found experimentally.

[0098] For each active interval, a meta-feature vector is defined, such as Figure 5The meta-feature vector encapsulates the relevant information about the activity during the active interval into a fixed-size vector. This vector is then used to detect or classify the activity.

[0099] For each feature f i , calculate a set of N statistical measures. By applying the function to each feature f i Examples of statistical measures are (but are not limited to): statistical moments (mean, variance, skew, kurtosis, etc.) and ordered statistics (minimum, maximum, median, quartiles, trimmed mean, median absolute deviation, etc.).

[0100] With Y features, a number of N×Y statistical measures are computed and encoded into a meta-feature vector for each active interval. This meta-feature represents the input to the activity detector. Figure 5 An example of a set of statistical measures for a feature group with four features is shown in FIG. The meta-vector is given by Figure 4A and Figure 4B Each of the active intervals shown is formed. Figure 5 As shown, a vector is formed by the mean of each feature in the interval, the variance of each feature in the interval, and the skew of each feature. Additional features, such as kurtosis, may be incorporated.

[0101] Based on the meta-feature vector, the decision unit then determines whether an alarm should be raised. The decision unit compares the determined meta-feature vector with a predefined set of vectors or classification models and detects anomalies based on the comparison. Initially, the classifier classifies the input data into defined output categories based on a mathematical model obtained through a supervised learning process. As part of the learning process, the mathematical model is provided with input pairs to obtain corresponding output data. Therefore, the model will represent a set consisting of a set of meta-features, a set of use cases, and the correspondence between the meta-features and the set of use cases. Based on the feedback as described above, the model is adjusted to maximize the classification accuracy by minimizing the error function of the model.

[0102] Figure 1 More details on the processing units are given in Figure 6 Described in.

[0103] Within the processing unit, two types of models are used for classification: a running model or first model 601 and a shadow or second model 602. Meta feature vectors as described above are input to the running model 601, the shadow model 602 and one or more backup models 605a-m. The running model 601 represents a model or predefined metadata vector for active decision making. During the above classification, this model is frozen, i.e., cannot be changed.

[0104] To maintain system robustness, shadow models 602 are developed in parallel and are based on operational model decisions made as well as operator or system feedback.

[0105] Feedback is provided to the validation and analysis module 603, which is also responsible for the exchange of models as described further below. It is worth noting that the shadow model has no influence on the decision. However, the shadow model is used to validate the decisions made using the running model.

[0106] To validate the decision, a comparison is made between the results of the shadow model and the results of the running model.

[0107] When the shadow model is superior to the running model, the verification, analysis and model exchange module 603 exchanges the shadow model with the running model. The shadow model becomes the running model. The replaced running model is saved as a backup model 605a-m. The backup model is a previously run module stored in the backup model buffer. The backup model can be restored to the running model as described below. The verification, analysis and model exchange module can be implemented by one or more processors.

[0108] The performance of the shadow model is f Each time feedback is received from the operator 604, the validation, analysis, and model exchange modules update the shadow model and evaluate the performance of the shadow model and the running model. The evaluation is based on N f The percentage of invalid decisions in cycles. f Varies depending on the desired accuracy of the system. Smaller N f A larger number N will lead to faster adaptation. f The higher the number, the higher the reliability at the expense of speed. f Sample numbers include: (a) for faster adaptability, it can be considered equal to feedback collected in 1 day to 1 week, and (b) for greater reliability, a value larger than feedback collected in 1 month can be chosen.

[0109] Consider three different situations arising from performance evaluation:

[0110] In the first case, the running model remains active. The shadow model is updated based on the received feedback (meta-features of the active intervals added to the shadow model’s training set). The shadow model is trained using the updated training set.

[0111] In the second case, when the performance of the shadow model exceeds that of the running model, the running model is swapped with the shadow model. The running model is stored in the backup model buffer, and the shadow model becomes the running model.

[0112] In the third case, the running model is swapped with the backup model. This happens when the backup model is better than the running model.

[0113] The backup model is saved in a buffer using a last most recently used mechanism. It should be appreciated that maintaining a backup model provides flexibility to cope with changes in the retail environment (e.g., changes in lighting, time of day, etc.). The system is thus optimized. Classification starts from the optimal point and uses the best performing model. Switching between the backup model and the running model is easier and more efficient than training a new model.

[0114] Once anomalies are classified using the running model as described above, alert(s) are transmitted as outlined above.

[0115] The combination of the above disclosed features provides an always-updatable model that improves anomaly detection in self-checkout retail environments through a consistent and replicable detection, training, and optimization process as described above.

[0116] The words "comprising" and the words "having / including" when used herein with reference to the present invention are used to specify the presence of stated features, integers, steps or components, but do not exclude the presence or addition of one or more other features, integers, steps, components or groups thereof.

[0117] It should be understood that, for the sake of clarity, certain features of the present invention described in the context of separate embodiments may also be provided in combination in a single embodiment. Conversely, for the sake of brevity, various features of the present invention described in the context of a single embodiment may also be provided separately or in any suitable sub-combination.

Claims

1. A system for anomaly detection in a self-service checkout environment, comprising: A processing unit configured to: extracting a set of features from transaction data received from a self-checkout terminal, the transaction data comprising video frames, and wherein the transaction data is associated with a self-checkout transaction at the self-checkout terminal; Characterize the activity based on the set of features; A number of active intervals are defined for each represented activity; determining a meta-feature vector for each defined activity interval in the plurality of activity intervals; Comparing each meta-feature vector with a set of predefined vectors or classification models, wherein the classification model represents a set consisting of a set of meta-features, a set of use cases, and a correspondence between the meta-features and the set of use cases; as well as detecting anomalies based on the comparison, Characterizing the activity includes: extracting a set of features from the received transaction data, converting the set of features into a corresponding set of feature activation values ​​and determining the evolution of the set of feature activation values ​​within a time interval, and defining the activity Activity according to the following formula μ : {f} i (n)|i∈featureSet μ ,μ∈[1,K],n∈[N1,N2],N2=N1+Na}, Where N a is the number of frames in which the activity is considered to be detected, K is the total number of activities recorded in the system knowledge base, f i (n) is the value of the extracted feature group, featureSet μ Describes the Activity μ A set of characteristics, [N1, N2] represents a set of consecutive frames extracted in the time interval between the time of the initial frame N1 and the time of the last frame N2, wherein the processing unit is further configured to estimate the number of frames defining the activity, and wherein the estimating comprises defining an activation function A(f i (n) f R i ) and divide the time domain into active intervals based on the activation function, and where A(f i (n) f R i ) is characterized as Where R i Represents the user for each feature f i A set of rules defined; when R i When true, f i (n) is active, where for all activities, each active interval is defined as Determining the meta-feature vector includes: for each feature f in the active interval i A number of statistical measures are calculated, and the features are combined to form a meta-feature vector for each active interval. 2 . The system of claim 1 , wherein the processing unit is further configured to determine an inactive interval between adjacent active intervals. 3 . The system of claim 2 , wherein the processing unit is further configured to compare the inactive interval with a threshold value, and combine the adjacent active interval and the inactive interval into a merged active interval. 4 . The system according to claim 1 , wherein the processing unit is configured to perform a first comparison between the determined meta-feature vector and a first classification model and to create an alert based on the first comparison. 5 . The system according to claim 4 , wherein the processing unit is configured to perform a second comparison between the determined meta-feature vector and the second classification model.

6. The system according to claim 5, wherein the processing unit is configured to compare the performance of the first classification model and the second classification model based on the first comparison and the second comparison, and replace the first classification model with the second classification model if the performance of the second classification model is better than the first classification model.

7. The system according to claim 6, further comprising: A memory for storing the second classification model as a backup model. 8 . The system of claim 7 , wherein the processing unit is configured to receive feedback on the alert and update the second classification model based on the feedback.

9. The system of claim 7, wherein the processing unit is configured to compare the performance of the first classification model with the performance of the backup model, and replace the first classification model with the backup model if the performance of the backup model is better than the first classification model.

10. A method for anomaly detection in a self-checkout environment, comprising: extracting a set of features from transaction data received from a self-checkout terminal, the transaction data comprising video frames, and wherein the transaction data is associated with a self-checkout transaction at the self-checkout terminal; Characterize the activity based on the set of features; A number of active intervals are defined for each represented activity; determining a meta-feature vector for each defined activity interval in the plurality of activity intervals; Comparing each meta-feature vector with a set of predefined vectors or classification models, wherein the classification model represents a set consisting of a set of meta-features, a set of use cases, and a correspondence between the meta-features and the set of use cases; as well as detecting anomalies based on the comparison, Characterizing the activity includes: extracting a set of features from the received transaction data, converting the set of features into a corresponding set of feature activation values ​​and determining the evolution of the set of feature activation values ​​within a time interval, and defining the activity Activity according to the following formula μ : {f} i (n)|i∈featureSet μ ,μ∈[1,K],n∈[N1,N2],N2=N1+N a }, Where N a is the number of frames in which the activity is considered to be detected, K is the total number of activities recorded in the system knowledge base, f i (n) is the value of the extracted feature group featureSet μ Describes the Activity μ A set of characteristics, [N1, N2] represents a set of consecutive frames extracted in the time interval between the time of the initial frame N1 and the time of the last frame N2, Wherein the method further comprises estimating the number of frames defining the activity, and wherein the estimating comprises defining an activation function A(f i (n) f R i ) and divide the time domain into active intervals based on the activation function, and where A(f i (n) f R i ) is characterized as Where R i Represents the user for each feature f i A set of rules defined; when R i When true, f i (n) is active, where for all activities, each active interval is defined as Determining the meta-feature vector includes: for each feature f in the active interval i A number of statistical measures are calculated, and the features are combined to form a meta-feature vector for each active interval.

11. A computer-readable programmable medium storing a computer program, which, when executed by a processor, implements the method according to claim 10.

Images