Behavior recognition method and system based on commodity state change, medium and equipment

Abstract

The invention provides a behavior recognition method and system based on commodity state change, a medium and equipment. The method comprises the following steps: acquiring a to-be-identified video, and performing perspective correction on each frame of to-be-identified image in the to-be-identified video to generate a standard view angle image; extracting a lattice image corresponding to each predetermined lattice area from the standard view angle image; for each lattice area, calculating a commodity state feature of the lattice area in the standard view angle image of the current frame and a commodity state feature sequence obtained in continuous frames, and calculating and outputting an event classification result of each lattice area; and performing cross-lattice consistency correction on the different lattice areas, and obtaining a behavior recognition result of the user based on a correction result. According to the method, the user behavior is identified by focusing the state change of the commodity side, the dependence on the user identity and the human body posture is avoided, and accurate monitoring can be completed without deploying a complicated multi-camera network or extra weight and RFID sensors.

Description

Technical Field

[0001] This application belongs to the field of computer vision processing technology, and relates to a behavior recognition method, system, medium and device based on changes in the state of goods. Background Technology

[0002] Traditional retail customer behavior recognition relies heavily on video images, typically requiring the detection and tracking of customers themselves, using technologies such as multiple cameras, facial / body recognition, or pose estimation. Unmanned retail solutions, exemplified by Amazon Go, employ sophisticated methods combining computer vision, sensor fusion, and deep learning to achieve a "grab and go" experience.

[0003] In real-world shopping, customer behavior is highly unpredictable: they don't always pick up an item and buy it; they often pick it up, look at it, and then put it back, sometimes even placing it in the wrong shelf. This makes video-based behavioral judgment extremely complex, requiring precise differentiation of subtle actions such as "picking up" and "putting down."

[0004] Existing solutions come at a high cost—for example, some systems deploy infrared curtain sensors in front of shelves to detect hands reaching in, then use cameras combined with human posture recognition to determine the action of picking up goods; at the same time, multiple cameras are installed to locate which item was taken, and multiple customers are tracked using the Mask R-CNN algorithm. Another example is self-service shopping solutions like AiFi, which use a large number of cameras to cover the store space; this not only results in redundant deployment but also requires solving occlusion problems and incurring huge computational demands for video processing.

[0005] Another approach integrates shelf weight sensors and cameras: a typical system embeds multiple weighing units in each shelf compartment, combines them with human key point tracking from the camera above, and uses sensor fusion algorithms to associate weight change events with specific customers.

[0006] However, these multimodal solutions heavily rely on human pose recognition to match weight events with customers, which can easily lead to confusion and association errors when multiple people interact or pick up goods across different locations. In addition, vision solutions often face challenges such as occlusion and interference from similar packaging, requiring expensive deep learning models, while sensor solutions such as weight or RFID have limitations in accuracy (it is difficult to detect weight changes in small items) and high deployment costs. Summary of the Invention

[0007] This application provides a behavior recognition method, system, medium, and device based on changes in product status, which solves the problem that multimodal solutions for recognizing customer behavior rely heavily on human posture, leading to confusion and association errors during multi-person interaction or cross-location product retrieval.

[0008] To achieve the above and other related objectives, a first aspect of this application provides a behavior recognition method based on changes in the state of a commodity. The method includes: acquiring a video to be recognized, and performing perspective correction on each frame of the video to generate a standard viewpoint image;

[0009] Extract the grid images corresponding to the predetermined grid regions from the standard viewpoint image;

[0010] For each grid region, calculate its commodity status features in the standard viewpoint image of the current frame;

[0011] For each grid region, calculate the product state feature sequence formed by the product state features calculated in consecutive frames, and calculate and output the event classification result for each grid region;

[0012] Cross-grid consistency correction is performed on different grid regions, and the user's behavior recognition result is obtained based on the correction result.

[0013] In some embodiments of the first aspect of this application, perspective correction is performed on each frame of the image to be identified in the video to be identified to generate a standard viewpoint image, including:

[0014] Obtain the coordinates of multiple original vertices and multiple target vertices of the image to be identified in each frame;

[0015] Calculate the homography matrix based on the coordinates of the original vertices and the coordinates of the target vertices;

[0016] Using the homography matrix, a perspective transformation is performed on the image to be identified to generate a standard viewpoint image.

[0017] In some embodiments of the first aspect of this application, the product state feature includes one or more combinations of the following features: pixel occlusion rate feature, bounce feature, pose proxy feature, specular phase feature, and optical flow feature.

[0018] In some embodiments of the first aspect of this application, a sequence of product state features is calculated from the product state features obtained by each grid region in consecutive frames, and an event classification result for each grid region is calculated and output, including:

[0019] The temporal pattern representation hidden in the product state feature sequence is extracted using a temporal coding model;

[0020] The temporal pattern representation is input into a classification network to generate confidence scores corresponding to different states or events;

[0021] Output the event classification results for the grid region based on the confidence level.

[0022] In some embodiments of the first aspect of this application, the process of extracting the temporal pattern representation using the temporal coding model and generating confidence scores by the classification network is implemented through a unified temporal classification model;

[0023] The time-series classification model is configured to: read a fixed-length feature sequence and output the prediction confidence for a preset category;

[0024] The temporal classification model is a Transformer encoder, which includes at least one encoder layer. Each encoder layer contains a multi-head self-attention mechanism and a feedforward neural network. The classification network is a feedforward network connected after the output layer of the Transformer encoder.

[0025] In some embodiments of the first aspect of this application, the method further includes:

[0026] Detect whether there are transient, minute perturbations in the product state feature sequence;

[0027] Based on the detected brief and minute disturbance, it is determined that a light touch event exists in the corresponding grid area, and the state classification of the corresponding grid area based on the product state feature sequence is stopped; wherein, the product state feature sequence corresponding to the brief and minute disturbance meets a preset amplitude threshold and decay characteristics in time.

[0028] In some embodiments of the first aspect of this application, cross-cell consistency correction is performed on different cell regions, including:

[0029] Based on preset state rules and / or syntax constraints, the event classification results at different times and in different grid regions are correlated and checked for consistency.

[0030] When the event sequence or the number of spanned cells is found to be inconsistent with the state rules and / or the grammatical constraints, the event classification result is corrected to obtain a consistent behavior recognition result.

[0031] To achieve the above and other related objectives, a second aspect of this application provides a behavior recognition system based on changes in the state of goods. The system includes:

[0032] The preprocessing module is configured to acquire the video to be identified and perform perspective correction on each frame of the image to be identified in the video to generate a standard viewpoint image.

[0033] The grid extraction module is configured to extract grid images corresponding to predetermined grid regions from the standard viewpoint image;

[0034] The feature extraction module is configured to calculate the commodity state features in the standard view image of the current frame for each grid region.

[0035] The classification module is configured to, for each grid region, calculate the product state feature sequence formed by the product state features calculated in consecutive frames, and calculate and output the event classification result for each grid region.

[0036] The recognition result output module is configured to perform cross-grid consistency correction on different grid regions and obtain the user's behavior recognition result based on the correction result.

[0037] In some embodiments of the second aspect of this application, the system further includes:

[0038] The touch detection module is configured to detect whether there are brief, minute disturbances in the product state feature sequence; and

[0039] It is configured to determine that a light touch event exists in the corresponding grid area based on the detected brief and minute perturbation, and stop classifying the state of the corresponding grid area based on the product state feature sequence;

[0040] The commodity state feature sequence corresponding to the brief and minute perturbation satisfies a preset amplitude threshold and decay characteristics in time.

[0041] To achieve the above and other related objectives, a third aspect of this application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the behavior recognition method based on changes in commodity status as described in any of the first aspects.

[0042] To achieve the above and other related objectives, a fourth aspect of this application provides an electronic device, including a memory and a processor;

[0043] The memory is configured to store computer programs;

[0044] The processor is configured to execute a computer program stored in the memory to enable the electronic device to implement the behavior recognition method based on changes in the state of goods as described in any of the first aspects.

[0045] As described above, the behavior recognition method, system, medium, and device based on changes in commodity status described in this application have the following beneficial effects:

[0046] This application identifies customer behavior by focusing on changes in the state of the goods, avoiding reliance on customer identity and posture, and eliminating the need to deploy complex multi-camera networks or additional weight and RFID sensors to achieve accurate monitoring.

[0047] The video to be identified in this application only involves the location and movement of goods, with zero exposure to customer personal information, thus meeting stringent data protection requirements. Furthermore, because it utilizes only video input from a single camera and performs intelligent analysis locally, the entire system can be lightweightly deployed on edge devices without uploading sensitive video streams to the cloud, reducing bandwidth and cloud computing costs while further ensuring privacy and security.

[0048] In terms of accuracy and robustness, this application achieves high-sensitivity detection of subtle behaviors such as picking up / putting down, misplacing, and light touching by multi-source feature fusion and temporal deep modeling. Compared with solutions that rely on pure visual tracking or pure weight sensing, this application utilizes rich visual features and rule-constrained cross-validation events, which can significantly improve the accuracy of event recognition in complex scenarios, especially maintaining stable output in situations with multiple interactions and frequent occlusion.

[0049] It has good versatility: it does not rely on prior knowledge of specific products or specific store layouts (no need for precise shelf plan calibration). With simple camera calibration and a small number of sample learning, it can be applied to different product types and shelf formats. Compared with traditional inventory monitoring that requires manual maintenance of product lists and display diagrams, it can automatically adapt to the addition and change of products, reducing operation and maintenance costs.

[0050] Since it does not require complex backend infrastructure and centralized computing, this application can be easily scaled up for large-scale deployment: retailers can install low-power smart cameras on each shelf to achieve real-time behavior collection and analysis of the entire store. This architecture is both cost-effective and efficient, and is expected to be widely used in the intelligent upgrade of physical retail.

[0051] This application will bring direct value in improving the user shopping experience (such as timely discovery of customer interests) and optimizing store operations (such as analyzing why products are viewed but not purchased). Attached Figure Description

[0052] Figure 1 The diagram shown is a schematic representation of an implementation scenario of the behavior recognition method based on changes in the state of goods described in this application.

[0053] Figure 2a and Figure 2b The diagram shown is a flowchart illustrating the behavior recognition method based on changes in product status as described in this application embodiment.

[0054] Figure 3 The image shown is a specification view of the shelf as described in an embodiment of this application.

[0055] Figure 4 The diagram shows the occlusion of the grid area as described in the embodiments of this application.

[0056] Figure 5 The diagram shows the relationship between displacement amplitude and time for the rebound characteristics described in this application embodiment.

[0057] Figure 6 The diagram shows a flowchart illustrating the calculation of commodity state feature sequences and commodity event classification results as described in the embodiments of this application.

[0058] Figure 7 The diagram shown is a structural schematic of the behavior recognition system based on changes in the state of goods as described in an embodiment of this application.

[0059] Figure 8 The diagram shown is a structural schematic of the electronic device described in an embodiment of this application.

[0060] Component designation explanation

[0061] 11 cell phone 12 Tablet PC 13 laptop 70 Behavior recognition system based on changes in product status 71 Preprocessing module 72 Grid Extraction Module 73 Feature extraction module 74 Classification module 75 Recognition result output module 76 Touch detection module 8 electronic devices 81 processor 82 memory 821 operating system 822 app 83 Network interface 84 bus system 85 User Interface S21~S25, S24' step S241~S243 step Detailed Implementation

[0062] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0063] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this application. Therefore, the drawings only show the components related to this application and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0064] In applications of customer behavior recognition in retail environments, some solutions deploy infrared curtain sensors in front of shelves to detect hands reaching in, and then use cameras combined with human posture recognition to determine the actions of picking up goods; other solutions integrate shelf weight sensors and cameras, embed multiple weighing units in the shelves according to the grid, and combine them with human key point tracking by the camera above, using sensor fusion algorithms to associate weight change events with specific customers.

[0065] However, the above methods mostly rely on human posture recognition to match weight events with customers. If there are multiple people interacting or picking up goods at different locations in the scenario, confusion and association errors are likely to occur. In addition, in practical applications, vision solutions often face challenges such as occlusion and interference from similar packaging. To handle these problems, expensive deep learning models are required, while sensor solutions such as weight or RFID have limitations in accuracy (it is difficult to detect weight changes of small items) and high deployment costs.

[0066] To address the aforementioned issues, this application provides a method, system, medium, and device for behavior recognition based on changes in the state of goods. Instead of directly tracking customers, it analyzes and identifies customer behavior through changes in the state of the goods themselves, using information such as the movement of the goods' position and whether they are obstructed to provide feedback on customer actions. By focusing on changes in the state of the goods, this application avoids reliance on user identity and posture, and eliminates the need for complex multi-camera networks or additional weight and RFID sensors to achieve accurate monitoring.

[0067] The behavior recognition method based on changes in commodity state in this application can be applied to, for example... Figure 1 The electronic devices shown in this application may include mobile phones 11 with wireless charging capabilities, tablet computers 12, laptop computers 13, wearable devices, in-vehicle devices, augmented reality (AR) / virtual reality (VR) devices, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), etc. The specific types of electronic devices are not limited in this application embodiment.

[0068] For example, electronic devices can communicate with networks and other devices wirelessly. The aforementioned wireless communication can use any communication standard or protocol, including but not limited to Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Messaging Service (SMS), BT, GNSS, WLAN, NFC, FM, and / or IR technologies. The technical solutions in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0069] The principles and implementation methods of the behavior recognition method, system, medium, and device based on changes in commodity status according to the present application will be described in detail below with reference to the accompanying drawings, so that those skilled in the art can understand the behavior recognition method, system, medium, and device based on changes in commodity status according to the present application without creative effort.

[0070] To facilitate understanding of the embodiments of this application, a detailed description will first be provided with reference to FIG2. As shown in FIG2, the behavior recognition method based on changes in the state of goods in this application embodiment includes the following steps S21 to S25.

[0071] Step S21: Obtain the video to be identified, and perform perspective correction on each frame of the image to be identified in the video to generate a standard viewpoint image.

[0072] In this embodiment, the monitoring cameras of the monitored shelves are first calibrated and corrected to generate a standard view of the shelves. Then, the four-point correspondence of the shelf plane is obtained through camera calibration, and the plane homography matrix is ​​calculated to perform perspective transformation on each frame of the image to be identified in the video to be identified. This transforms the tilted shelf image from the original monitoring camera viewpoint into a standard perspective image of a top-down / frontal view, so that a fixed rectangular area is formed in the grid image on the shelf.

[0073] For example, perspective correction is performed on each frame of the image to be identified in the video to generate a standard viewpoint image, including: obtaining multiple original vertex coordinates and multiple target vertex coordinates of each frame of the image to be identified; calculating a homography matrix based on the multiple original vertex coordinates and multiple target vertex coordinates; and using the homography matrix to perform perspective transformation on the image to be identified to generate a standard viewpoint image.

[0074] The standard view image obtained after the above correction process can eliminate the scale and perspective distortion caused by the viewpoint, which facilitates the subsequent positioning and analysis of each cell.

[0075] In one embodiment, such as Figure 3 As shown, the video to be identified captured by the camera is processed by the grid standard view generation module and converted into a shelf top view that is easy to analyze.

[0076] In this embodiment, the camera is installed above and opposite the shelf, at a certain downward angle. A 3×3 homography matrix is ​​calculated by calibrating four corner points on the shelf plane. .

[0077] Among them, homography matrix The calculation formula is:

[0078] in, The coordinates of each frame in the video to be identified captured by the camera. The coordinates mapped to the standard viewpoint. Each element is calculated from four pairs of points, where the four pairs of points are the four corner points of each frame of the video to be identified and the four corner points of the standard view plane.

[0079] For example, the four corner points of each frame in the video to be identified are: top left (320, 180), top right (960, 200), bottom left (300, 720), and bottom right (980, 740). The four corner points of the standard viewing plane are: top left (0, 0), top right (640, 0), bottom left (0, 480), and bottom right (640, 480).

[0080] The homography matrix can be calculated using OpenCV's cv2.findHomography or cv2.getPerspectiveTransform. .

[0081] The following is the code implementation corresponding to the above embodiments:

[0082] import cv2

[0083] import numpy as np

[0084] src_pts = np.float32([[320,180],[960,200],[300,720],[980,740]]) # Original image four points

[0085] dst_pts = np.float32([[0,0],[640,0],[0,480],[640,480]]) # Target rectangle four points

[0086] H, status = cv2.findHomography(src_pts, dst_pts) # Calculate the homography matrix

[0087] print("Homography matrix H:\n", H)

[0088] Finally, the homography matrix is ​​obtained by calculation. The image to be recognized is subjected to perspective transformation to generate a standard viewpoint image.

[0089] Specifically, the warpPerspective(frame, H) transformation is performed to map the image to be recognized onto the shelf plane coordinate system to obtain a standard view image.

[0090] Step S22: Extract the grid images corresponding to the predetermined grid regions from the standard view image.

[0091] In this embodiment, a grid is a product placement unit on the shelf. Each grid corresponds to one product placement unit on the shelf. The grid area can be pre-divided according to the shelf layout, or the grid boundaries can be determined by marking. For example, each shelf layer is divided into 10 grids, and there are a total of 20 grids for two layers.

[0092] Specifically, based on the layout of the shelves, the geometric layout of each grid in the standard view image is predefined, and corresponding grid template data is established. Each grid unit corresponds to a grid area. The grid template data includes: the total number of grid units and the boundary coordinate information of each grid unit in the standard view image.

[0093] Based on the grid template data, a corresponding binary mask image is generated for each grid region. The mask image of the grid region to be extracted is obtained. The image data within the grid region is extracted by pixel-level bitwise AND operation or array index operation. Then, the extracted image data is reassembled into a standard rectangular image and normalized to obtain the extracted grid images corresponding to the predetermined grid regions.

[0094] The two-dimensional mask image has the same size as the standard view image; optionally, the normalization process includes adjusting the extracted rectangular images to a format with a fixed width and a fixed height.

[0095] Step S23: For each grid region, calculate its commodity status features in the standard view image of the current frame.

[0096] In this embodiment, multi-dimensional product status features are extracted for each grid area to encode the current status and trend of the product.

[0097] In this embodiment, the product status features include one or more of the following features in combination: pixel occlusion rate features, bounce features, pose proxy features, specular phase features, and optical flow features.

[0098] For example, calculating the pixel occlusion rate feature of the grid area in the standard view image of the current frame includes: recording the reference appearance of the product in the grid area using background modeling or an initial image to obtain a reference image, and calculating the proportion of the grid area being occluded by comparing the standard view image of the grid area in the current frame with the reference image.

[0099] Specifically, during the initialization phase, multiple frames of images of the grid area under unobstructed conditions are acquired. A reference image is established for each pixel position using a Gaussian mixture model or median filtering method. The standard view image of the grid area in the current frame is compared pixel by pixel with the reference image. By calculating Mahalanobis distance or Euclidean distance and setting an adaptive threshold, foreground occluded pixels and background pixels are distinguished. The basic occlusion rate is obtained by statistically analyzing the proportion of foreground occluded pixels to the total number of pixels.

[0100] Optionally, such as Figure 4 As shown, it can detect whether a customer's hand or other object is encroaching on the front of the product in the grid area, causing obstruction. The degree of obstruction is quantified by the ratio of the visible area after obstruction to the total area of ​​the grid area.

[0101] In this embodiment, if the occlusion rate is 100% and the occlusion continues for several frames, it usually means that the product has been taken away. If it is partially occluded, it may correspond to the customer's touch or occlusion viewing action before picking it up.

[0102] The rebound feature is used to represent the change in the displacement amplitude of an item in consecutive frames.

[0103] For example, calculating the rebound features of a grid region in the standard viewpoint image of the current frame includes: at the texture level, calculating the Barcol distance between the local binary pattern histogram and the established reference image, fitting a curve showing an exponential decay of this Barcol distance over time, and extracting the recovery rate constant; at the color level, tracking the decay process of the mean ΔE color difference in the Lab color space; and at the structural level, monitoring the structural similarity recovery curve between the edge image and the background edge image. By identifying a multi-stage pattern of fast recovery, slow recovery, and stable phases, and quantifying the duration and transition nodes of each stage, a multi-dimensional feature vector describing the dynamic characteristics of background recovery is finally formed.

[0104] In real-world scenarios, when a customer lightly touches or moves a product, the product may experience slight shaking or displacement and bounce back to its original position after the customer releases their grip. By analyzing the minute oscillations of key points on the product over time, a motion curve similar to damped vibration can be obtained. The rebound characteristics can reveal slight disturbances to the product. When making subsequent touch judgments, if the product exhibits a small instantaneous displacement that decays rapidly in the calculated rebound characteristics, it can be determined as a light touch rather than picking it up.

[0105] like Figure 5 As shown, a line graph represents the displacement of a product over time: at a certain key time point, the product has a small instantaneous displacement, and the displacement amplitude rapidly decreases after that key time point. At this time, the process of the product's displacement amplitude changing over time can be used to determine whether the customer has touched the product.

[0106] Posture proxy features are used to represent changes in the orientation or placement angle of an item.

[0107] For example, calculating the pose proxy features of the grid region in the standard viewpoint image of the current frame includes: obtaining a preliminary pose estimate by extracting the contour of the item and calculating the aspect ratio and rotation angle of its minimum bounding rectangle; simultaneously, analyzing the position and direction of the shadow generated by the item, and indirectly inferring the object's height and orientation by combining prior light source information; at the texture level, analyzing the dominant direction of the surface texture through gradient orientation histograms, and fitting an elliptical model using the spatial distribution of corner points or SIFT feature points to obtain its principal axis direction. By integrating these contour, shadow, texture, and geometric features, and tracking their evolution across multiple frames, a proxy feature vector capable of representing the spatial orientation and pose changes of the item is constructed.

[0108] In real-world scenarios, the degree of forward / side-turning of the proxies can be estimated by tracking angular changes in prominent patterns and / or edge lines on the product packaging within the grid area, or by using 3D pose estimation. Optionally, the visible area ratio of the product's front can be detected to infer whether the product's orientation has changed.

[0109] Highlight phase features are used to represent the distribution or intensity of highlights (specular reflection spots) on a product surface over time.

[0110] For example, calculating the highlight phase features of a grid region in the standard viewpoint image of the current frame includes: first, identifying highly saturated and bright specular reflection regions in the HSV color space, and tracking the temporal changes in their area, average brightness, and centroid position; performing a Fourier transform on the brightness time series, analyzing its power spectrum to extract the main frequency components and harmonics synchronized with the light source period; determining the synchronicity of their reflection changes by calculating the cross-correlation and phase difference of brightness signals between multiple regions; simultaneously, analyzing the envelope of the brightness signal to obtain the modulation depth and frequency information caused by the curvature change of the object surface; finally, compensating by comparing with the reference signal of the environmental background region to obtain a set of frequency domain and phase features closely related to the object surface material, shape, and relative illumination orientation.

[0111] In real-world scenarios, if a product is picked up and rotated, its highlight position often moves or disappears as the angle changes. By analyzing the position and brightness changes of the highlight area on the product's surface in consecutive frames, we can capture information about the change in the product's posture relative to the light source when it is rotated.

[0112] Optical flow features are used to describe the movement of pixels within a grid area and to estimate pixel velocity vectors between adjacent frames.

[0113] For example, calculating the optical flow features of a grid region in the standard viewpoint image of the current frame includes: for two consecutive grayscale images, using the Farneback dense optical flow or Lucas-Kanade sparse optical flow algorithm, calculating the two-dimensional motion vector of each pixel or feature point. Based on this vector field, extracting statistics such as average motion intensity, dominant motion direction, and directional consistency. Further analyzing the spatial distribution pattern of the motion vectors, identifying specific motion patterns such as inward convergence, outward diffusion, or rotation, and calculating the accumulation and change trend of motion energy within the time window, thereby forming a set of dynamic feature descriptors capable of distinguishing the entry, exit, or internal motion of objects.

[0114] In real-world scenarios, when an item is picked up, an upward light flow will appear in the corresponding grid area; when an item is put down, a downward light flow will appear in the corresponding grid area.

[0115] The following is the code implementation for steps S22 to S23 in the above embodiment:

[0116] initialize feature_sequence = {slot: [] for slot in grid_slots} # Create a data structure for the product status feature for each divided grid area.

[0117] for each frame in video_stream:

[0118] norm_view = warpPerspective(frame, H) # Perspective transformation

[0119] for each slot in grid_slots:

[0120] region = norm_view[ grid_masks[slot] ] # Extract grid region image

[0121] occ_ratio = compute_occlusion(region) # Calculate the occlusion rate

[0122] motion = compute_optical_flow(region) # Calculate the average displacement of the optical flow

[0123] orient = estimate_orientation(region) # Orientation proxy, such as the outer frame angle

[0124] highlight = measure_specular_shift(region) # Specular phase shift

[0125] features = [occ_ratio, motion, orient, highlight]

[0126] feature_sequence[slot].append(features)

[0127] In this embodiment, the standard viewpoint image of consecutive frames is processed in a loop using the above code, and the product status features of each cell are stored sequentially into the data structure of the created product status features.

[0128] The `compute_occlusion` function is used to obtain the proportion of occluded pixels through background subtraction or image differencing, thus obtaining pixel occlusion rate features. The `compute_optical_flow` function is used to calculate the average motion vector using the optical flow fields of adjacent frames, thus obtaining optical flow features. The functions `estimate_orientation` and `measure_specular_shift` are used to extract changes in the tilt angle of the product and changes in the position of surface highlights, thus obtaining pose proxy features and highlight phase features.

[0129] Step S24: For each cell region, calculate the product state feature sequence formed by the product state features calculated in consecutive frames, and calculate and output the event classification result for each cell region.

[0130] In this embodiment, the product state features composed of one or more of the pixel occlusion rate features, bounce features, pose proxy features, specular phase features, and optical flow features of consecutive frames are encoded into a product state feature sequence to continuously reflect the changes in the product state within the grid area over time. By fusing one or more of the feature information of pixel occlusion rate features, bounce features, pose proxy features, specular phase features, and optical flow features, the visual cues left by customers when operating on the products can be comprehensively depicted, providing rich basis for subsequent temporal modeling.

[0131] In real-world scenarios, since customer behavior has a significant temporal continuity, time series modeling can be used to analyze the product state feature sequence in order to identify characteristic behavioral patterns.

[0132] In some implementations, such as Figure 6 As shown, calculating the commodity status feature sequence and commodity event classification results includes the following steps S241 to S243.

[0133] Step S241: Use a temporal coding model to extract the temporal pattern representation hidden in the product state feature sequence.

[0134] In this embodiment, the process of extracting temporal pattern representations using a temporal coding model and generating confidence scores by a classification network is implemented through a unified temporal classification model. The temporal classification model is configured to read a fixed-length feature sequence and output the predicted confidence scores for a preset category. The temporal classification model is a Transformer encoder, which includes at least one encoder layer. Each encoder layer contains a multi-head self-attention mechanism and a feedforward neural network. The classification network is a feedforward network connected after the output layer of the Transformer encoder.

[0135] The Transformer encoder can model long-range dependencies through a self-attention mechanism, enabling it to focus on changes in the relationship between consecutive frames of a product's state, and effectively handle inconsistencies in the duration of behavior between consecutive frames. In practical operation, considering the limited computing resources of edge devices, a lightweight Transformer structure can be preferred, reducing the number of encoder layers or hidden dimensions while ensuring real-time operation.

[0136] The temporal classification model built by the Transformer encoder introduces a temporal dimension, which can identify fine-grained behavioral fragments for subsequent accurate differentiation of various actions.

[0137] Alternatively, the above temporal classification can also be achieved by using temporal convolutional networks (TCN) or hidden Markov models (HMM).

[0138] Temporal convolutional networks are used: one-dimensional convolution is used to extract features on the time axis. Causal convolution and dilated convolution can be used to efficiently capture temporal patterns. The number of parameters is small, making it suitable for long sequence modeling.

[0139] Hidden Markov Models are used: the state of a commodity is discretized into several hidden states (such as "still", "lightly moved", "picked up", "removed", etc.), and the observed sequence is explained by learning the state transition probability. Hidden Markov Models have advantages for small data and clear rules, but their disadvantage is that they are difficult to express complex sequence features.

[0140] Step S242: Input the temporal pattern representation into the classification network to generate confidence scores corresponding to different states or events.

[0141] Step S243: Output the event classification results of the grid area based on the confidence level.

[0142] In this embodiment, the event classification results of the output grid area include, but are not limited to: pick up, put down, misplaced, light touch, obstructed view, close inspection and rotation, etc.

[0143] Among them, "pick up" means the customer takes the product away, "put down" means the customer puts the product back in its original position, "place in the wrong place" means the customer puts the product back in a non-original compartment, "touch lightly" means the customer only touches the product briefly without doing anything else, "cover up to view" means the customer moves / picks up the product to view it, thus covering the product behind it, and "examine closely and rotate" means the customer picks up the product and rotates it in their hand to examine it.

[0144] In this embodiment, a multi-classification model based on neural networks is preferably used, and the temporal pattern representation hidden in the commodity state feature sequence extracted from step S241 above is used as input.

[0145] To balance the relevance and differences of different events, the structure of a multi-classification model can adopt multi-head output or hierarchical classification: for example, first distinguish whether there is a picking-up action, and then, under the premise of picking up, determine whether to put it back in its original position, put it back in a misplaced position, or not put it back. Alternatively, the light touch and picking up and putting down can be classified in parallel, and then a comprehensive decision can be made.

[0146] Step S25: Perform cross-grid consistency correction on different grid areas, and obtain the user's behavior recognition result based on the correction result.

[0147] It should be noted that some events appear similar visually. For example, both normal return and misplacement involve picking up and putting down, only the placement position is different.

[0148] To this end, in addition to utilizing the temporal features of a single cell, the multi-classification model can also refer to the state changes of adjacent cells. For example, when determining whether a placement is a "misplacement," it can check whether the appearance of the placed item matches the item originally belonging to that cell, or whether another cell is missing an item at the same time. For occlusion viewing events, the pattern is usually that the front-row item is briefly moved a short distance and then put back without leaving the field of view. This can be distinguished from a true pick-up by the magnitude and duration of the change in the item's occlusion rate (in occlusion viewing, the item does not completely leave the frame and quickly returns to its position). Closer inspection of rotation shows that the item is picked up and remains in the air for a relatively long time with significant posture changes (such as changes in highlight distribution), before finally being put back. These complex patterns are all automatically identified by the multi-classification model based on pre-learned rules. Furthermore, the multi-classification model output can include the temporal boundaries of the event (start-end frames), facilitating further applications.

[0149] The multi-classification model of this application enables the system to monitor multiple types of customer-product interaction behaviors simultaneously, providing a more refined ability to distinguish behaviors compared to traditional methods that only detect pick-up / put-down events.

[0150] In this embodiment, cross-cell consistency correction is performed on different cell regions, including: based on preset state rules and / or grammatical constraints, performing association and consistency checks on event classification results at different times and in different cell regions; when it is found that the event sequence or cross-cell number changes do not conform to the state rules and / or grammatical constraints, the event classification results are corrected to obtain consistent behavior recognition results.

[0151] Syntactic constraints only constrain logical rules governing the order of events, and can be compared to state machines or regular grammars. For example, logically, an item cannot simply disappear or appear without a corresponding pick-up or put-down event. Therefore, the logical rules are set as follows: if an item disappears from a certain cell but no pick-up event is detected, the events within that consecutive frame need to be re-evaluated as pick-up times; if a new item suddenly appears in a cell that was not originally placed there, a misplacement event should be identified accordingly; furthermore, for a complete interaction, the syntactic order of pick-up followed by put-down should be satisfied. Any pick-up that ends before being put down should be marked as "out-of-store purchase," and conversely, put-down should not occur without prior pick-up.

[0152] To implement the above grammatical constraints, embodiments of this application define state rules, which include normal state sequence processing strategies and abnormal state sequence processing strategies. The normal state sequence processing strategies sequentially include: leave still, pick up, remove / view, and put back / leave the store.

[0153] For example, if the event classification results obtained are in an abnormal order of "idle" and "placed" in chronological order, it can be determined that the product placed in this grid area is not the original product of this grid area. Therefore, it is necessary to correct the event classification result to be a misplacement of products in other grid areas.

[0154] In this embodiment, the sequence of event classification results output by the multi-classification model is parsed a second time using grammatical constraints. Using a method similar to context-free grammar, possible event sequences are matched with defined legal patterns. If the sequence deviates from the legal grammar, it is adjusted and completed based on the closest legal sequence.

[0155] Optionally, a simplified rule engine is constructed to encode various constraints in if-else form to check the event classification results output by the multi-classification model one by one. For example, when it is detected that the event classification result of a certain grid area includes the event "picked up and not returned", and the event classification result of another grid area includes the event "put down (misplacement of goods in other grid areas)", and the time matches, the event classification results of the two grid areas can be associated, and "picked up and not returned" and "put down (misplacement of goods in other grid areas)" can be merged and interpreted as a single "cross-grid misplacement" event. The merged event is then labeled with the source grid area where the goods were mistakenly taken and the target grid area where the goods were misplaced.

[0156] By correcting the aforementioned constraint rules or conditions, inconsistencies caused by misclassification in multi-classification models or environmental noise can be effectively eliminated. For example, transient false detections caused by occlusion can be identified and filtered by rules; temporary chaotic states caused by multiple customers operating simultaneously can also be reasonably decomposed through grammatical constraints. Through the above process, the final output event sequence can be ensured to be logically consistent and reliable, significantly improving the robustness of the system.

[0157] In some implementations, such as Figure 2b As shown, the behavior recognition method based on changes in commodity status in this application further includes step S24'.

[0158] Step S24': Detect whether there is a brief, minor disturbance in the product state feature sequence; based on the detected brief, minor disturbance, determine that there is a light touch event in the corresponding grid area, and stop classifying the grid area based on the product state feature sequence.

[0159] Among them, the commodity state feature sequence corresponding to the brief and minute perturbation meets the preset amplitude threshold and decay characteristics in time.

[0160] In this embodiment, a "perturbation-damping" model is introduced for detection and judgment of the special subtle behavior of light touch.

[0161] Specifically, a brief touch to a product is considered a pulse perturbation. When a customer touches a product, it experiences a small displacement or vibration, which gradually stops after the customer releases the touch due to the damping effect of the shelf and the product itself. By analyzing the product state characteristic sequence in the target compartment area, a typical pattern of a pulse perturbation can be fitted. For example, in the rebound characteristic, the curve corresponding to a light touch shows: the displacement suddenly increases but the amplitude does not exceed a threshold, and then returns to rest in an exponential decay manner.

[0162] Optionally, the optical flow characteristics in the product state feature sequence can be analyzed to detect and determine that the optical flow characteristics have a brief non-zero vector at the moment of a touch, and then the non-zero vector rapidly approaches zero. During the touch detection process, if a "peak-attenuation" pattern is detected in the optical flow characteristics of the product state feature sequence and the overall change amplitude is small, it is determined that a touch has occurred.

[0163] Optionally, the position or angle of the product in consecutive frames can be fitted to calculate the attenuation ratio; if it conforms to the damped vibration model (such as the amplitude dropping to a certain proportion below the initial value within several frames), a light touch event can be output.

[0164] The perturbation-damping model effectively distinguishes between unintentional touches and actual picking up, enabling the system to detect interactions when customers merely flick the product with their fingers without actually picking it up. This is significant for analyzing customer interests and improving product display. Furthermore, combining the touch detection results with the main time-series model output and considering them together in the final classification stage improves sensitivity to subtle behaviors and recognition accuracy.

[0165] The following is the code implementation for steps S24 to S25 and step S24' in the above embodiment:

[0166] events = {} # Events used to store the judgment of each cell position

[0167] for each slot in grid_slots:

[0168] seq = feature_sequence[slot] # Retrieves the accumulated feature sequence for this slot

[0169] # 1. Touch detection: Determines whether there are transient, minute perturbation patterns in the sequence.

[0170] if detect_light_touch(seq):

[0171] events[slot] = "touch"

[0172] continue # Output the touch event separately, without further checking for pick-up and put-down.

[0173] # 2. Temporal Model Inference: Obtaining Preliminary Classification Results

[0174] pred = temporal_model_infer(seq) # For example, the output is {"pick up":0.01, "put down":0.0, "pick up and put down":0.85, ...}

[0175] # Select the event category with the highest probability as the initial result.

[0176] event_label = argmax(pred)

[0177] events[slot] = event_label

[0178] # 3. Apply rules / syntactic constraints to perform cross-cell consistency correction

[0179] for each slot, event in events:

[0180] # If a item is picked up but not returned, check if there is a corresponding misplacement.

[0181] if event == "Picked up unreturned":

[0182] for each other_slot, other_event in events:

[0183] if other_event == "Mistakenly Returned": # Mistaken events were found in other cells

[0184] # Misplaced cases where the two are marked as related

[0185] events[slot] = "Pick up (misplaced source)"

[0186] events[other_slot] = "Mistakenly placed (from slot %s)" % slot

[0187] # If an isolated drop event occurs and there was no prior picking up, you can adjust the settings to include obscuring the view, etc.

[0188] if event == "put down" and no_prior_pick(slot):

[0189] events[slot] = "Obstructs view"

[0190] In this embodiment, the temporal modeling and event classification module is used to perform behavior recognition on the accumulated product state feature sequence. For each grid area, the temporal model first extracts the pattern of its product state feature sequence.

[0191] Specifically, a two-layer Transformer encoder is used as the temporal model, with its hidden dimension and number of self-attention heads pruned to adapt to the computing power of embedded devices. The Transformer model reads a feature sequence vector of length N and outputs a hidden representation sequence of the same length. Subsequently, the last hidden representation is fed into a feedforward classification network to generate the confidence scores of events occurring within the time window.

[0192] Simultaneously, touch detection is performed to detect the amplitude and decay trend of movement in the product state feature sequence. If it is found that the product only shakes slightly within a certain period of time and quickly returns to stillness, a touch event signal is recorded.

[0193] Finally, the output of the classification network, along with the touch signal and the association information of adjacent cells, is fed into the multi-event decision logic to determine the final event category.

[0194] The code above first checks the light touch event one grid at a time. If the detect_light_touch function determines that the light touch is based on the slight displacement and rapid decay of the product, then the event of that grid is directly defined as "light touch", and subsequent event classification is skipped (because there is no product being picked up in the case of a light touch).

[0195] Next, the temporal model is invoked via temporal_model_infer to analyze the product state feature sequence and obtain a preliminary event prediction distribution. The category with the highest confidence is then taken as the result. For example, if the prediction result for a certain grid area indicates that the probability of "picking up and putting down" is 85%, then the event is tentatively defined as "picking up and putting down" (i.e., picking up and then putting back).

[0196] Next comes the rule correction phase: The code above demonstrates the application of two syntax rules—first, if an event in a certain cell is classified as "picked up but not returned" (meaning the item was taken away and not returned by the end of the observation window), then other cell areas are searched for events classified as "misplaced and returned" (meaning an item was placed in a non-original location); if a match is found, it is determined that the two events correspond to the same customer's misplacement behavior: the event in the original cell is marked as "picked up (misplacement source)", and the event in the target cell is marked as "misplaced (from cell X)", thus clearly linking the two together for easier subsequent analysis; similarly, for cases where "put down" occurs but there is no corresponding "picked up" beforehand, the rule engine will interpret it as occlusion rather than a true return action (because it is physically impossible to put down without picking up, the actual meaning should be that the item was moved and then put back in its original place).

[0197] The no_prior_pick(slot) function is used to check whether the slot has previously experienced a product being removed from the shelf in the time series. If not, it means that this "pick-up" is unusual and is more likely to be a case of obstructed viewing.

[0198] The above code flow enables automatic identification and reasonable attribution of various complex events. It's worth noting that in practical implementation, more rules can be added as needed, such as restricting the possibility of two customers simultaneously taking the same item, or limiting the likelihood of a customer repeatedly picking up and putting down the same item within a very short period. These prior knowledge points can be transformed into syntactic constraints and incorporated into the rule engine, improving the reliability of the system's output.

[0199] The scope of protection of the behavior recognition method based on changes in commodity status in this application is not limited to the execution order of the steps listed in this embodiment. Any solution implemented by adding, subtracting, or replacing steps in the prior art based on the principles of this application is included within the scope of protection of this application.

[0200] This application also provides a behavior recognition system based on changes in product status. The behavior recognition system based on changes in product status can implement the behavior recognition method based on changes in product status of this application. However, the implementation device of the behavior recognition method based on changes in product status of this application includes, but is not limited to, the structure of the behavior recognition system based on changes in product status listed in this embodiment. All structural modifications and substitutions of the prior art made in accordance with the principles of this application are included within the protection scope of this application.

[0201] Please see Figure 7 This is shown as a behavior recognition system based on changes in the state of goods in an embodiment of this application.

[0202] like Figure 7As shown, the behavior recognition system 70 based on changes in product status includes:

[0203] The preprocessing module 71 is configured to acquire the video to be recognized and perform perspective correction on each frame of the image to be recognized in the video to generate a standard viewpoint image.

[0204] The grid extraction module 72 is configured to extract grid images corresponding to predetermined grid regions from a standard viewpoint image;

[0205] The feature extraction module 73 is configured to calculate the commodity status features in the standard view image of the current frame for each grid region.

[0206] The classification module 74 is configured to calculate, for each cell region, the sequence of commodity state features formed by the commodity state features calculated in consecutive frames, and to calculate and output the event classification result for each cell region.

[0207] The recognition result output module 75 is configured to perform cross-grid consistency correction on different grid areas and obtain the user's behavior recognition result based on the correction result.

[0208] Optionally, it also includes a touch detection module 76, configured to detect whether there is a brief, minor perturbation in the product state feature sequence; and

[0209] It is configured to determine that a light touch event exists in the corresponding grid area based on the detected brief and minute perturbation, and stop classifying the state of the corresponding grid area based on the product state feature sequence.

[0210] Among them, the commodity state feature sequence corresponding to the brief and minute perturbation meets the preset amplitude threshold and decay characteristics in time.

[0211] Based on the presence of a brief, minor disturbance, it is determined that a light touch event has occurred in the grid area, and the state classification of the grid area based on the commodity state feature sequence is stopped.

[0212] It should be understood that the module division in the embodiments of this application is illustrative and only represents a logical functional division. In actual implementation, there may be other division methods. Furthermore, the functional modules in the various embodiments of this application can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0213] In the embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, or methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules / units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or units may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of apparatuses or modules or units may be electrical, mechanical, or other forms.

[0214] The modules / units described as separate components may or may not be physically separate. The components shown as modules / units may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules / units can be selected to achieve the objectives of the embodiments of this application, depending on actual needs. For example, the functional modules / units in the various embodiments of this application may be integrated into one processing module, or each module / unit may exist physically separately, or two or more modules / units may be integrated into one module / unit.

[0215] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0216] Figure 8 This is a schematic block diagram of the electronic device provided in an embodiment of this application. Figure 8 As shown, the electronic device 800 includes at least one processor 81, a memory 82, at least one network interface 83, and a user interface 85. The various components in the device are coupled together via a bus system 84. It is understood that the bus system 84 is used to implement communication between these components. In addition to a data bus, the bus system 84 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 8 The general will label all buses as bus systems.

[0217] The user interface 85 may include a monitor, keyboard, mouse, trackball, clicker, button, touchpad, or touch screen.

[0218] It is understood that memory 82 can be volatile memory or non-volatile memory, or both. Among them, non-volatile memory can be read-only memory (ROM) or programmable read-only memory (PROM), which is used as an external cache.

[0219] In this embodiment, the memory 82 is used to store various types of data to support the operation of the electronic device 80. Examples of such data include any executable programs that operate on the electronic device 80, such as the operating system 821 and application programs 822; the operating system 821 contains various system programs, such as the framework layer, core library layer, driver layer, etc., for implementing various basic services and handling hardware-based tasks.

[0220] The methods disclosed in the embodiments of this application described above can be applied to processor 81, or implemented by processor 81. Processor 81 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by integrated logic circuits in the hardware of processor 81 or by instructions in software form.

[0221] This application also provides a computer-readable storage medium. Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing a processor. The program can be stored in a computer-readable storage medium, which is a non-transitory medium, such as random access memory, read-only memory, flash memory, hard disk, solid-state drive, magnetic tape, floppy disk, optical disk, and any combination thereof. The storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital video disc (DVD)), or a semiconductor medium (e.g., solid-state drive (SSD)).

[0222] This application embodiment may also provide a computer program product comprising one or more computer instructions. When the computer instructions are loaded and executed on a computing device, all or part of the processes or functions described in this application embodiment are generated. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0223] When the computer program product is executed by a computer, the computer performs the method described in the foregoing method embodiments. The computer program product can be a software installation package; when the foregoing method is required, the computer program product can be downloaded and executed on the computer.

[0224] The descriptions of the processes or structures corresponding to the above figures each have their own emphasis. For parts of a process or structure that are not described in detail, please refer to the relevant descriptions of other processes or structures.

[0225] In summary, this application provides a method, system, medium, and device for behavior recognition based on changes in the state of goods. By focusing on changes in the state of the goods to identify customer behavior, it avoids reliance on customer identity and body posture, eliminating the need for complex multi-camera networks or additional weight and RFID sensors to achieve accurate monitoring. Through multi-source feature fusion and temporal deep modeling, it achieves high-sensitivity detection of subtle behaviors such as picking up / putting down, misplacing, and light touching. Compared to solutions relying on pure visual tracking or pure weight sensing, this application utilizes rich visual features and rule-constrained cross-validation events, which can significantly improve the accuracy of event recognition in complex scenarios, especially maintaining stable output under conditions of multi-person interaction and frequent occlusion. Therefore, this application effectively overcomes the various shortcomings of existing technologies and has high industrial application value.

[0226] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.

Claims

1. A behavior recognition method based on changes in the state of goods, characterized in that, The method includes: The video to be identified is acquired, and perspective correction is performed on each frame of the image to be identified in the video to generate a standard viewpoint image. Extract the grid images corresponding to the predetermined grid regions from the standard viewpoint image; For each grid region, calculate its commodity status features in the standard viewpoint image of the current frame; For each grid region, calculate the product state feature sequence formed by the product state features calculated in consecutive frames, and calculate and output the event classification result for each grid region; Cross-grid consistency correction is performed on different grid regions, and the user's behavior recognition result is obtained based on the correction result.

2. The behavior recognition method based on changes in commodity status according to claim 1, characterized in that, Perspective correction is performed on each frame of the image to be identified in the video to generate a standard viewpoint image, including: Obtain the coordinates of multiple original vertices and multiple target vertices of the image to be identified in each frame; Calculate the homography matrix based on the coordinates of the original vertices and the coordinates of the target vertices; Using the homography matrix, a perspective transformation is performed on the image to be identified to generate a standard viewpoint image.

3. The behavior recognition method based on changes in commodity status according to claim 1, characterized in that, The product status features include one or more of the following features in combination: pixel occlusion rate features, bounce features, pose proxy features, specular phase features, and optical flow features.

4. The behavior recognition method based on changes in commodity status according to claim 1, characterized in that, Calculate the product state feature sequence formed by the product state features obtained by each grid region in consecutive frames, and calculate and output the event classification result for each grid region, including: The temporal pattern representation hidden in the product state feature sequence is extracted using a temporal coding model; The temporal pattern representation is input into a classification network to generate confidence scores corresponding to different states or events; The event classification results for the grid region are output based on the confidence level.

5. The behavior recognition method based on changes in commodity status according to claim 4, characterized in that, The process of extracting the temporal pattern representation using the temporal coding model and generating confidence scores using the classification network is achieved through a unified temporal classification model. The time-series classification model is configured to: read a fixed-length feature sequence and output the prediction confidence for a preset category; The temporal classification model is a Transformer encoder, which includes at least one encoder layer. Each encoder layer contains a multi-head self-attention mechanism and a feedforward neural network. The classification network is a feedforward network connected after the output layer of the Transformer encoder.

6. The behavior recognition method based on changes in commodity status according to claim 1, characterized in that, The method further includes: Detect whether there are transient, minute perturbations in the product state feature sequence; Based on the detected brief and minute perturbation, it is determined that a light touch event exists in the corresponding grid area, and the state classification of the grid area based on the product state feature sequence is stopped. The commodity state feature sequence corresponding to the brief and minute perturbation satisfies a preset amplitude threshold and decay characteristics in time.

7. The behavior recognition method based on changes in commodity status according to claim 1, characterized in that, Perform cross-cell consistency correction on different cell regions, including: Based on preset state rules and / or syntax constraints, the event classification results at different times and in different grid regions are correlated and checked for consistency. When the event sequence or the number of spanned cells is found to be inconsistent with the state rules and / or the grammatical constraints, the event classification result is corrected to obtain a consistent behavior recognition result.

8. A behavior recognition system based on changes in the state of goods, characterized in that, The system includes: The preprocessing module is configured to acquire the video to be identified and perform perspective correction on each frame of the image to be identified in the video to generate a standard viewpoint image. The grid extraction module is configured to extract grid images corresponding to predetermined grid regions from the standard viewpoint image; The feature extraction module is configured to calculate the commodity state features in the standard view image of the current frame for each grid region. The classification module is configured to, for each grid region, calculate the product state feature sequence formed by the product state features calculated in consecutive frames, and calculate and output the event classification result for each grid region. The recognition result output module is configured to perform cross-grid consistency correction on different grid regions and obtain the user's behavior recognition result based on the correction result.

9. The behavior recognition system based on changes in commodity status according to claim 8, characterized in that, The system also includes: The touch detection module is configured to detect whether there are brief, minor disturbances in the product state feature sequence; and It is configured to determine that a light touch event exists in the corresponding grid area based on the detected brief and minute perturbation, and to stop classifying the corresponding grid area based on the product state feature sequence; The commodity state feature sequence corresponding to the brief and minute perturbation satisfies a preset amplitude threshold and decay characteristics in time.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the behavior recognition method based on changes in commodity status as described in any one of claims 1 to 7.

11. An electronic device, characterized in that, Including memory and processor; The memory is configured to store computer programs; The processor is configured to execute a computer program stored in the memory to enable the electronic device to implement the behavior recognition method based on changes in commodity status as described in any one of claims 1 to 7.

Images