High-value commodity off-shelf state tracking method, device and system based on computer vision

Abstract

The invention provides a high-value commodity off-shelf state tracking method based on computer vision, belongs to the field of computer vision driven intelligent retail security, and aims to distinguish high-value commodities and reduce false detection and missing detection by constructing a high-precision three-dimensional space coordinate library and establishing a binding relation between the commodities and coordinates. And the interaction behavior between the personnel and the commodity is analyzed and judged in combination with space-time correlation, so that the accuracy of action recognition is ensured. According to the method, the accuracy of commodity tracking and abnormal behavior alarm is effectively improved, and the method is suitable for a retail security scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of machine vision, and in particular to a high-value commodity off-shelf state tracking system and method based on computer vision. BACKGROUND

[0002] In the supermarket loss prevention scene, the loss prevention of high-value commodities has always been an important challenge for retailers. Traditional loss prevention measures mainly rely on manual monitoring, simple alarm devices, and some physical protection methods, which have many limitations.

[0003] Firstly, manual monitoring is easily affected by factors such as fatigue, negligence, and limited monitoring range, and cannot real-time and comprehensively pay attention to every corner, especially in a supermarket environment with high customer flow. Simple alarm devices have a high false alarm rate and cannot provide accurate positioning and real-time tracking information, which greatly reduces their effectiveness in practical applications. Although physical protection methods can improve the security of high-value commodities to some extent, they bring inconvenience to customers and affect the shopping experience, and cannot fundamentally solve the problem of commodity theft.

[0004] With the continuous development of computer vision technology, its application in the security field has gradually attracted attention. Existing commodity tracking technologies based on computer vision mostly focus on the analysis of customer behavior or the monitoring of specific areas, but these technologies still have deficiencies in high-value commodity loss prevention. For example, some systems can only issue an alarm when the commodity leaves a specific area, cannot track the dynamic trajectory of the commodity in real time, and cannot identify and bind the personnel interacting with the commodity, making it difficult to quickly locate and track suspects after the commodity is stolen. In addition, existing technologies often cannot effectively distinguish high-value commodities from non-high-value commodities within the same shelf, resulting in waste of monitoring resources and inability to accurately focus monitoring on high-value commodities.

[0005] In terms of risk area identification and handling, existing technologies also have defects. The monitoring blind spots and dead angles in supermarkets are high-risk areas for theft, but traditional monitoring systems often cannot identify these areas in advance and cannot effectively track and record personnel and commodities entering these areas. Once the commodity is hidden in the monitoring dead angle, it is difficult to effectively investigate and handle the theft due to the lack of a complete evidence chain.

[0006] Therefore, the present application aims to propose a high-value commodity off-shelf state tracking system and method based on computer vision, which realizes the binding of high-value commodities and personnel to be tracked by constructing a spatial coordinate library of high-value commodities, accurately distinguishes high-value commodities from non-high-value commodities, and identifies and handles monitoring blind spots and dead angles in advance, thereby improving the loss prevention efficiency of high-value commodities, optimizing the allocation of monitoring resources, and improving the monitoring efficiency. SUMMARY

[0007] Embodiments of the present application provide a high-value commodity off-shelf state tracking method, device and system based on computer vision. The technical solution is as follows:

[0008] In one aspect, a high-value commodity off-shelf state tracking method based on computer vision is provided, which is realized by an electronic device. The method comprises: constructing a high-value commodity initial position three-dimensional space coordinate library, and establishing a binding relationship between high-value commodity uniqueness and three-dimensional space coordinates;

[0009] Detecting hand movements of personnel entering the high-value area, and when the hand movement trajectory is spatially coupled with the displacement of the commodity, extracting the personnel's physical features and marking them as personnel to be tracked;

[0010] Marking high-value commodities that are not returned within a first threshold time as commodities to be tracked, and establishing a binding relationship between the commodities to be tracked and the personnel to be tracked;

[0011] When the movement trajectory of the personnel or commodities to be tracked is predicted to enter a high-risk area by a path planning algorithm, activate the cross-camera tracking module;

[0012] If the commodity to be tracked has not completed payment and meets the following conditions: entering a high-risk area or leaving a high-value area for more than a second threshold time, send an early warning message.

[0013] Preferably, the early warning message includes the clothing texture feature vector of the personnel to be tracked, the cross-camera movement trajectory heat map, and the category information, off-shelf time axis data and last known coordinates of the commodity to be tracked.

[0014] Preferably, the coordinate data is dynamically updated by an automatic calibration mechanism:

[0015] When the shelf is physically displaced, coordinate mapping is updated based on feature point matching of the calibration plate;

[0016] When the commodity is moved, spatial interpolation compensation is performed based on the relative positions of adjacent commodities.

[0017] Preferably, the high-risk area determination method comprises: constructing a risk heat map based on historical theft data, marking the monitoring blind area as a region; using an LSTM model to predict the target movement trajectory, and triggering an early warning when the probability of the trajectory endpoint entering the risk area exceeds 70%.

[0018] Preferably, the cross-camera tracking comprises: deploying AprilTag calibration plates in the overlapping field of view of adjacent cameras, calculating homography matrix to realize coordinate unification; constructing a multi-modal ReID feature vector, including: clothing texture features extracted by improving ResNet-34, with SE attention module added to the last layer of the network; step frequency features calculated based on foot key point trajectories, with dynamic time warping (DTW) algorithm used to match gait cycles; and HSV color histogram features, with polarized lenses used to suppress glare interference.

[0019] Preferably, the hierarchical calibration comprises:

[0020] Deploy AprilTag calibration plates with different IDs on each layer of the shelf, and calculate the tilt angle compensation coefficient of each layer through perspective transformation;

[0021] Predefine ROI dynamic adjustment strategy based on the size of the commodity packaging: when adjacent commodities are removed, automatically expand the current ROI coverage to 80% of the empty area of the shelf.

[0022] Preferably, it further comprises secondary verification of interactive actions:

[0023] When the Pearson correlation coefficient is greater than 0.7, detect the duration of the closed state of the hand:

[0024] Calculate the normalized distance from the fingertips to the palm root through the MediaPipe hand key point model

[0025] If the distance is less than 30% of the palm width for more than 3 frames, it is determined to be a grasping state. Only when the correlation coefficient condition and the grasping state are both met at the same time, the effective interaction is confirmed.

[0026] Preferably, the fusion method of the multi-modal ReID feature vector comprises:

[0027] Min-Max normalization is performed on the clothing texture features, step frequency features, and color histogram respectively;

[0028] A dynamic weighted fusion strategy is adopted:

[0029] When the light is uniform, the weight distribution is 4:3:3

[0030] Under low light conditions, adjust to 2:5:3

[0031] The fused feature vector is matched across the mirror through cosine similarity, with a threshold of 0.65.

[0032] Preferably, the ROI dynamic adjustment strategy further comprises: when the shelf layer height changes by more than ±10%, triggering an inter-layer matching algorithm based on SIFT feature points; extracting SIFT feature descriptors of the current layer and the reference layer; removing false matching points by RANSAC algorithm; and updating the ROI coordinates according to the affine transformation matrix calculated from the matching points.

[0033] Preferably, the hand closed state detection comprises an anti-interference mechanism: when the hand occlusion is detected to be more than 50% of the area, motion continuity verification is enabled; the acceleration change rate of the hand motion trajectory of the previous 5 frames is analyzed; if the acceleration standard deviation is less than 0.2 m / s 2 and the direction angle change is less than 15°, the original state is maintained; otherwise, a multi-view verification process of the adjacent camera is started.

[0034] In another aspect, a computer vision-based high-value commodity off-shelf state tracking system is provided. The device is applied to a computer vision-based high-value commodity off-shelf state tracking method. The system comprises:

[0035] A coordinate library construction module is configured to construct a high-value commodity initial position three-dimensional space coordinate library and establish a binding relationship between high-value commodity uniqueness and three-dimensional space coordinates.

[0036] An action detection module is configured to detect hand actions of personnel entering a high-value area, extract personnel physical features when the hand motion trajectory and the commodity displacement form a spatial coupling, and mark the personnel as a to-be-tracked personnel.

[0037] A commodity marking module is configured to mark high-value commodities that are not placed back within a first threshold time as to-be-tracked commodities, and establish a binding relationship between the to-be-tracked commodities and the to-be-tracked personnel.

[0038] A trajectory prediction module is configured to predict the motion trajectory of the to-be-tracked personnel or commodities, and activate a cross-camera tracking module when the trajectory enters a high-risk area.

[0039] A warning module is configured to send a warning information when the to-be-tracked commodities have not completed payment and meet the conditions of entering a high-risk area or leaving a high-value area for more than a second threshold time.

[0040] Preferably, the warning module is further configured to generate a warning information comprising a clothing texture feature vector of the to-be-tracked personnel, a cross-camera motion trajectory heat map, and category information, off-shelf time axis data, and last known coordinates of the to-be-tracked commodities.

[0041] Preferably, the coordinate library construction module comprises:

[0042] A calibration board deployment module is configured to deploy AprilTag calibration boards on the shelves.

[0043] Coordinate system establishment module: for establishing a shelf coordinate system through the feature points of the calibration board;

[0044] Coordinate conversion module: for converting the shelf coordinate system to the global coordinate system through the homography matrix;

[0045] Layered calibration module: for implementing layered calibration on commodities in the same shelf, setting an independent detection region (ROI) for high-value commodities, and controlling the coordinate accuracy within a range of ±5cm.

[0046] In another aspect, an electronic device is provided, which includes a processor and a memory having at least one instruction stored therein, the at least one instruction being loaded and executed by the processor to implement the above-mentioned computer vision-based high-value commodity off-shelf tracking method.

[0047] In another aspect, a computer-readable storage medium is provided, which has at least one instruction stored therein, the at least one instruction being loaded and executed by a processor to implement the above-mentioned computer vision-based high-value commodity off-shelf tracking method.

[0048] The technical solutions provided by the embodiments of the present application have at least the following beneficial effects:

[0049] 1. By constructing an initial position coordinate library of high-value commodities and using an AprilTag calibration board for accurate calibration, the system can accurately distinguish high-value commodities from ordinary commodities on the same shelf, avoiding the misjudgment and missed detection problems caused by the dense placement of commodities in traditional monitoring systems.

[0050] 2. By using the YOLOv5 hand detection model and spatiotemporal correlation analysis, combined with MediaPipe hand key point detection, the interactive action of personnel and commodities is accurately identified, reducing misjudgment, and combined with the first threshold time and the second threshold time, reducing the system alarm caused by commodities being placed randomly after being taken off the shelf, and improving the system alarm efficiency.

[0051] 3. When the commodity is taken up and not put back within the set time, it is automatically marked as a "to-be-tracked commodity" and bound with the personnel, forming a dynamic association, which is convenient for subsequent tracking and effectively deals with the behavior of commodity transfer or concealment. BRIEF DESCRIPTION OF DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0053] Figure 1It is a high-value commodity off-shelf state tracking method architecture schematic diagram provided by the embodiment of the application.

[0054] Figure 2 It is a high-value commodity off-shelf state tracking method logic flow chart provided by the embodiment of the application.

[0055] Figure 3 It is Figure 1 The coordinate library construction and information transmission timing diagram in the middle;

[0056] Figure 4 It is Figure 1 The action detection and personnel marking timing diagram in the middle;

[0057] Figure 5 It is Figure 1 The commodity marking timing diagram in the middle;

[0058] Figure 6 It is Figure 1 The trajectory prediction and risk judgment timing diagram in the middle;

[0059] Figure 7 It is Figure 1 The early warning decision timing diagram in the middle. DETAILED DESCRIPTION

[0060] In order to make the technical problems, technical solutions and advantages of the present application more clear, the following will be described in detail in combination with the drawings and specific embodiments.

[0061] The embodiment of the application provides a high-value commodity off-shelf state tracking method, device and system based on computer vision. The technical solution is as follows:

[0062] On the one hand, a high-value commodity off-shelf state tracking method based on computer vision is provided, which is realized by an electronic device, and the method comprises:

[0063] As shown in Figure 1 and Figure 2 , a high-value commodity initial position three-dimensional space coordinate library is constructed, and a high-value commodity uniqueness and three-dimensional space coordinate binding relationship is established; coordinate data is dynamically updated through an automatic calibration mechanism, when the shelf is physically displaced, coordinate mapping update is realized based on feature point matching of the calibration plate, and when the commodity is moved, spatial interpolation compensation is carried out through the relative position of adjacent commodities.

[0064] As shown in Figure 3 , wherein the construction of the high-value commodity initial position coordinate library comprises: establishing a shelf coordinate system O s -X s Y s Z s , and converting it to a global coordinate system Og X g Y g Z g The solution of homography matrix H is based on least squares method, for the point (u, v) on the image plane and the point (X, Y, Z) in the world coordinate system, there is the following relationship:

[0065]

[0066] Assuming that there are n sets of corresponding points in the calibration process, by constructing an overdetermined linear equation system a i ·h = 0 (where a i is composed of image points and world point coordinates, h is the vector form after the homography matrix H is expanded), singular value decomposition (SVD) is used to solve h, and then the homography matrix H is obtained.

[0067] The goods in the same shelf are implemented hierarchical calibration, the shelf is divided into m layers in the vertical direction, and the height of each layer is h i (i = 1, 2, …, m). For high-value goods, an independent detection area (ROI) is set, and the coordinate accuracy is controlled within ±5 cm. In actual calculation, the coordinate accuracy is improved by taking the average value through multiple measurements. Assuming that the coordinates of a certain high-value goods are measured k times, the coordinate values (x j , y j , z j )(j = 1, 2, …, k) are obtained, and the final coordinate value is:

[0068]

[0069] In summary, the goods in the same shelf are implemented hierarchical calibration, and the independent detection area (ROI) is set for high-value goods, and the coordinate accuracy is controlled within ±5 cm. It effectively solves the problem that ordinary monitoring cameras are difficult to distinguish similar adjacent goods, which easily leads to misidentification.

[0070] Further, when the shelf is physically displaced, coordinate mapping update is realized based on the feature point matching of the calibration board. The ORB feature point detection algorithm is used to extract the feature points on the AprilTag calibration board, the similarity between the feature points is calculated by Hamming distance, and the matched feature point pairs are found. Let P s1 (x s1 , y s1 , z s1 ) be the point in the original shelf coordinate system, and P s2 (x s2 , y s2 , z s2 ) be the corresponding point in the shelf coordinate system after displacement, and the coordinate conversion is realized by the similarity transformation matrix T:

[0071]

[0072] where the similarity transformation matrix T contains a rotation matrix R, a translation vector t and a scale factor s:

[0073]

[0074] The similarity transformation matrix T is solved by minimizing the re-projection error of the matched point pairs, the formula of the re-projection error e is:

[0075]

[0076] where u i is the observed point coordinate in the image, X i is the point coordinate in the world coordinate system, and π is the projection function.

[0077] Further, when the commodity is moved, the spatial interpolation compensation is performed through the relative positions of the adjacent commodities. Assuming that the initial coordinates of the moved commodity A are (x A0 ,y A0 ,z A0 ), the coordinates of the adjacent commodities B and C are (x B ,y B ,z B ) and (x C ,y C ,z C ) respectively. Using the triangular interpolation method, assuming that the coordinates of the moved commodity A are (x A ,y A ,z A ), then:

[0078] x A = x A0 + α (x B - x A0 ) + β (x C - x A0 )

[0079] y A = y A0 + α (y B - y A0 ) + β (y C - y A0 )

[0080] z A = z A0 + α (z B - z A0 ) + β (z C - z A0 )

[0081] wherein a and b are interpolation coefficients determined by minimizing the distance variation between the moving commodity and the adjacent commodity.

[0082] Further, a unique serial number is generated for each high-value commodity, and a high-density tamper-resistant two-dimensional code such as a QR Code is printed; for metal commodities, an anti-metal RFID tag is used.

[0083] As shown in Figure 4 , the hand movement of a person entering the high-value area is detected, and when the hand movement trajectory forms a spatial coupling with the commodity displacement, the physical features of the person are extracted and marked as a person to be tracked. The spatial coupling includes: extracting hand key points through a YOLOv5 hand detection model; establishing a spatio-temporal correlation model of commodity displacement vector and hand movement trajectory; when the Pearson correlation coefficient of the two movement vectors is > 0.7, it is determined to be an effective interaction.

[0084] Specifically:

[0085] 1. Hand detection and key point extraction: real-time detection of hand bounding box in monitoring screen, filtering non-hand area interference; extracting 21 hand key points such as fingertips, knuckles, and palm center; calculating hand opening posture or holding posture; converting hand key points from image coordinate system to global 3D coordinate system, which can adapt to different lighting and occlusion scenes, and reduce false detection. Determine whether the hand is in a grasping state (e.g., the distance between the fingertips and the palm center is < 30% of the palm width) by the key points to avoid misjudging "touch" as "pick up".

[0086] Converting hand key points from image coordinate system to global 3D coordinate system requires relying on AprilTag calibrated shelf coordinate system. Let the point (u, v) in the image coordinate system be converted to the point (X, Y, Z) in the global 3D coordinate system through the intrinsic matrix K and the extrinsic matrix [R|t]:

[0087]

[0088] wherein the intrinsic matrix K includes focal length f x ,f y and optical center coordinates c x ,c y :

[0089]

[0090] The extrinsic matrix [R|t] includes the rotation matrix R and the translation vector t. Determine whether the hand is in a grasping state by the key points, and when the distance between the fingertips and the palm center d < 0.3w (w is the palm width), it is determined to be in a grasping state.

[0091] 2. Spatiotemporal coupling analysis of commodity displacement and hand trajectory: Track the center point coordinates of high-value commodities through target detection (YOLOv5), calculate the displacement changes (Δx, Δy, Δz) between consecutive frames; Based on the coordinate changes of hand key points (such as palm point), generate motion trajectory; Correlation analysis is performed on the motion vectors (such as 10-frame displacement sequence) of commodities and hands, if PCC>0.7, correlation analysis is performed, and Pearson correlation coefficient (PCC) is used for calculation:

[0092]

[0093] where x i and y i are the displacements of the commodity and the hand in the i-th frame, and are the average values of the displacements of the commodity and the hand. When PCC>0.7, it is determined as "effective interaction" (strong correlation); otherwise, it is considered as irrelevant action.

[0094] 3. Personnel feature extraction and tracking marker: Extract clothing texture features through improved ResNet-34, in the network structure, SE module enhances the response of important features in the network by weighting the channel dimension. In terms of gait features, calculate the step frequency based on foot key point trajectory, and use DTW algorithm to match gait cycle. Let two gait sequences X={x1,x2,…,x m} and Y={y1,y2,…,y n}, the DTW algorithm measures the similarity by calculating the minimum cumulative distance D between the two sequences:

[0095] D(i,j)=d(x i ,y j )+min{D(i-1,j),D(i,j-1),D(i-1,j-1)}

[0096] where d(x i ,y j ) is the distance between the corresponding points in the two sequences.

[0097] 4. To improve the recognition accuracy, the interactive action is verified again, the clothing texture features are extracted through the improved ResNet-34 (adding SE attention module), in the network structure, SE module enhances the response of important features in the network by weighting the channel dimension. In terms of gait features, calculate the step frequency based on foot key point trajectory, and use DTW algorithm to match gait cycle. Let two gait sequences X={x1,x2,…,x m} and Y={y1,y2,…,y n}, the DTW algorithm measures the similarity by calculating the minimum cumulative distance D between the two sequences:

[0098] D(i,j)=d(x i ,y j )+min{D(i-1,j),D(i,j-1),D(i-1,j-1)}

[0099] where d(x i ,y j ) is the distance between corresponding points in the two sequences.

[0100] In summary, from "hand contact" to "product displacement" to "personnel binding", a closed-loop anti-theft logic is formed.

[0101] like Figure 5 As shown, the high-value goods that are not returned within the first threshold time are marked as goods to be tracked, and a binding relationship between the goods to be tracked and the people to be tracked is established;

[0102] Specifically:

[0103] 1. When a high-value item is picked up, the system starts a timer. If the item does not return to the initial ROI coordinates (±5cm range) within T1, it is marked as a "pending item." The Kalman filter predicts the item's trajectory, preventing coordinate loss caused by temporary occlusion. The system records the item's off-shelf timestamp, last known coordinates, and the ID of the person associated with the item, forming a structured log. Using in-memory databases such as Redis, the system implements millisecond-level status updates, supporting high-concurrency scenarios. This allows customers to view items normally (even if they are off the shelf for a short period of time), and only triggers tracking for items that have not returned to their original location after the timeout.

[0104] 2. Establishment of dynamic binding relationship between products and personnel

[0105] Product side: record category ID, appearance characteristics, and off-shelf trajectory;

[0106] On the personnel side: clothing texture, gait characteristics, and real-time behavior (whether they are moving toward high-risk areas) extracted when binding interactions occur;

[0107] Use a graph database (such as Neo4j) to store "person-product-time" triples. When the product is transferred, the binding relationship is updated through cross-camera ReID. Even if the product is hidden or transferred, the source can still be traced through the personnel characteristics.

[0108] like Figure 6 As shown, when the trajectory of the person or item being tracked enters a high-risk area, as predicted by the path planning algorithm, the cross-camera tracking module is activated. The high-risk area identification method includes: constructing a risk heat map based on historical theft data and marking blind spots; using an LSTM model to predict the target's trajectory; and triggering an alert when the probability of the trajectory's endpoint entering a risky area exceeds 70%.

[0109] Specifically:

[0110] The trajectory planning algorithm is to analyze the motion trajectory (continuous 10-frame coordinate sequence) of the target (to-be-tracked person / commodity) using an LSTM time series model, predict the probability distribution of the path endpoint in the next 5 seconds, and trigger tracking when the probability of the predicted endpoint falling into the fence of the GIS coordinate of the high-risk area (such as the monitoring blind area and the exit channel) is > 70%. The cross-border tracking method is to deploy AprilTag calibration plates in the overlapping field of view of adjacent cameras, calculate the homography matrix to realize coordinate unification, construct a multi-modal ReID feature vector, which includes clothing texture features extracted by improving ResNet-34, with an added SE attention module at the end of the network, step frequency features calculated based on foot key point trajectory, with a dynamic time warping (DTW) algorithm to match the gait cycle, and HSV color histogram features, with a polarizing lens to suppress glare interference. At the same time, in order to generate a cross-camera motion trajectory heat map, the system collects the motion trajectory data of the target in multiple cameras, including the target position coordinates of each frame, time-aligns the trajectory data to ensure their consistency in time, then maps the aligned trajectory data into a global coordinate system to form a global trajectory, calculates the density distribution of the global trajectory to generate a heat map, where each pixel value of the heat map represents the trajectory density at that position, and the density is represented by the color depth, and finally the heat map is superimposed on the floor plan of the supermarket to visually display the activity frequency and path of the target in different areas.

[0111] As Figure 7 shown, if the to-be-tracked commodity has not completed payment and meets the following conditions: entering a high-risk area or leaving a high-value area for more than a second threshold time, an early warning information is sent; the early warning information includes the clothing texture feature vector of the to-be-tracked person, the cross-camera motion trajectory heat map, and the category information of the to-be-tracked commodity, the off-shelf time axis data, and the last known coordinate. By setting clear trigger conditions, the critical state of commodity theft is accurately identified. The detailed early warning information covers multi-dimensional data of personnel, commodities, and trajectories, which not only helps security personnel quickly lock suspicious targets, but also provides complete evidence chain for post-investigation, realizing full-process coverage from real-time prevention to traceability processing.

[0112] Further, the hierarchical calibration includes: deploying different ID AprilTag calibration boards for each layer of the shelf, calculating the tilt angle compensation coefficient of each layer through perspective transformation; and predefining a ROI dynamic adjustment strategy based on the size of the commodity packaging: when adjacent commodities are removed, automatically expanding the current ROI coverage to 80% of the empty area of the shelf. The hierarchical calibration and the ROI dynamic adjustment solve the problems of shelf space diversity and commodity layout changes, and through accurate tilt angle compensation, ensure the unified calibration of commodity coordinates of different heights; and the ROI dynamic expansion mechanism can adapt to changes in shelf commodities, avoid monitoring blind spots caused by changes in commodity positions, and realize omnidirectional and dead-angle-free monitoring of high-value commodities.

[0113] Further, it further includes secondary verification of interactive actions: when the Pearson correlation coefficient is greater than 0.7, the duration of the closed state of the hand is detected: the normalized distance from the fingertip to the palm root is calculated through the MediaPipe hand key point model; if the distance is less than 30% of the palm width for more than 3 frames, it is determined to be a gripping state, and only when the correlation coefficient condition and the gripping state are met at the same time, the effective interaction is confirmed, the secondary verification mechanism greatly improves the reliability of the interactive action recognition, effectively filters the false positives caused by non-theft behaviors such as accidental touch of the hand and rapid swipe, reduces the false alarm rate of the system, ensures that each alarm corresponds to a real commodity taking risk, and improves the use efficiency of security resources.

[0114] Further, the fusion method of the multi-modal ReID feature vector includes: performing Min-Max normalization on the clothing texture feature, step frequency feature, and color histogram respectively; adopting a dynamic weighted fusion strategy: when the light is uniform, the weight distribution is 4:3:3; under low light conditions, it is adjusted to 2:5:3; the fused feature vector is matched across the mirror through cosine similarity, and the threshold is set to 0.65.

[0115] Further, the ROI dynamic adjustment strategy further includes: when the shelf height changes more than ±10%, triggering an inter-layer matching algorithm based on SIFT feature points: extracting SIFT feature descriptors of the current layer and the reference layer; removing the mis-matching points through the RANSAC algorithm; and calculating the affine transformation matrix according to the matching point pairs to update the ROI coordinates.

[0116] Further, the hand closed state detection includes an anti-interference mechanism: when the hand occlusion is detected to be more than 50% of the area, motion continuity verification is enabled: the acceleration change rate of the hand motion trajectory of the previous 5 frames is analyzed, if the acceleration standard deviation is less than 0.2m / s 2And the direction angle changes <15°, the original state is maintained, otherwise the adjacent camera multi-view verification process is started, the anti-interference mechanism effectively solves the hand shielding problem in complex scene, through multi-dimensional verification strategy, the interaction state can still be accurately judged in partial shielding, avoiding missing detection or misjudgment caused by shielding, and the reliability of the system in crowded shopping environment is ensured.

[0117] In another aspect, a computer vision-based high-value commodity off-shelf state tracking system is provided. The device is applied to a computer vision-based high-value commodity off-shelf state tracking method. The system comprises:

[0118] A coordinate library construction module is configured to construct a high-value commodity initial position three-dimensional space coordinate library and establish a high-value commodity uniqueness and three-dimensional space coordinate binding relationship.

[0119] An action detection module is configured to detect hand actions of personnel entering a high-value area, extract personnel physical features and mark as a to-be-tracked personnel when a hand movement trajectory is spatially coupled with a commodity displacement.

[0120] A commodity marking module is configured to mark a high-value commodity that is not placed back within a first threshold time as a to-be-tracked commodity and establish a to-be-tracked commodity and to-be-tracked personnel binding relationship.

[0121] A trajectory prediction module is configured to predict the movement trajectory of a to-be-tracked personnel or commodity, and activate a cross-camera tracking module when the trajectory enters a high-risk area.

[0122] A warning module is configured to send a warning information when a to-be-tracked commodity has not completed payment and meets entering a high-risk area or leaving a high-value area for more than a second threshold time.

[0123] Further, the warning module is further configured to generate a warning information including a to-be-tracked personnel's clothing texture feature vector, a cross-camera movement trajectory heat map, and a to-be-tracked commodity's category information, off-shelf time axis data, and last known coordinates.

[0124] Further, the coordinate library construction module comprises:

[0125] A calibration board deployment module is configured to deploy an AprilTag calibration board on a shelf.

[0126] A coordinate system establishment module is configured to establish a shelf coordinate system through calibration board feature points.

[0127] A coordinate conversion module is configured to convert the shelf coordinate system to a global coordinate system through a homography matrix.

[0128] A hierarchical calibration module is configured to implement hierarchical calibration of commodities within the same shelf, set an independent detection area (ROI) for high-value commodities, and control the coordinate accuracy within ±5cm.

[0129] In another aspect, an electronic device is provided, which includes a processor and a memory having stored therein at least one instruction, which is loaded and executed by the processor to implement the above computer vision based high value item out-of-cabinet tracking method.

[0130] In another aspect, a computer readable storage medium is provided, which has stored therein at least one instruction, which is loaded and executed by a processor to implement the above computer vision based high value item out-of-cabinet tracking method.

[0131] The above description is merely illustrative of the application and not restrictive thereof; it will be understood by those skilled in the art that modifications, improvements, equivalents, etc., made to the application, as long as they are within the spirit and principle of the application, shall be included in the scope of protection of the application.

Claims

1. A method for tracking the off-shelf status of high-value goods based on computer vision, characterized by: The method comprises: Build a three-dimensional spatial coordinate library of the initial positions of high-value goods, and establish a binding relationship between the uniqueness of high-value goods and their three-dimensional spatial coordinates; Detect hand movements of people entering high-value areas. When the hand movement trajectory forms a spatial coupling with the displacement of goods, the person's physical features are extracted and marked as a person to be tracked. High-value goods that are not returned within the first threshold time are marked as goods to be tracked, and a binding relationship is established between the goods to be tracked and the people to be tracked; When the trajectory of the person or product to be tracked enters a high-risk area as predicted by the path planning algorithm, the cross-camera tracking module is activated; If the payment for the product to be tracked has not been completed and meets the following conditions: it enters a high-risk area or leaves a high-value area for more than a second threshold time, an early warning message will be sent.

2. The method according to claim 1, characterized in that The warning information includes the clothing texture feature vector of the person to be tracked, the cross-camera motion trajectory heat map, and the category information, off-shelf timeline data and last known coordinates of the product to be tracked.

3. The method according to claim 1, characterized in that The construction of the high-value commodity initial position coordinate library includes: The shelf coordinate system is established using the AprilTag calibration plate and converted to the global coordinate system using the homography matrix; Layered calibration is implemented for goods on the same shelf, and independent detection areas (ROIs) are set up for high-value goods, with coordinate accuracy controlled within ±5cm.

4. The method according to claim 1, wherein Dynamically update coordinate data through the automatic calibration mechanism: When the shelf is physically displaced, the coordinate mapping is updated based on the feature point matching of the calibration plate; When a product is moved, spatial interpolation compensation is performed based on the relative positions of adjacent products.

5. The method according to claim 1, wherein The interactive action recognition includes: Extract hand key points through the YOLOv5 hand detection model; Establish a spatiotemporal correlation model between product displacement vector and hand motion trajectory; When the Pearson correlation coefficient of the two motion vectors is greater than 0.7, the effective interaction is determined.

6. The method according to claim 1, characterized in that The high-risk area determination method includes: constructing a risk heat map based on historical theft data and marking the monitoring blind areas; The LSTM model is used to predict the target's trajectory, and an early warning is triggered when the probability that the end point of the trajectory enters the risk area exceeds 70%.

7. The method according to claim 1, characterized in that The cross-border tracking includes: Deploy AprilTag calibration plates in the overlapping fields of view of adjacent cameras and calculate the homography matrix to unify coordinates. The multimodal ReID feature vector is constructed, which includes: clothing texture features extracted by improving ResNet-34, and adding a SE attention module to the last layer of the network; gait frequency features calculated based on the foot key point trajectory, and the dynamic time warping (DTW) algorithm is used to match the gait cycle; HSV color histogram features are used to suppress reflection interference using a polarized lens.

8. The method according to claim 3, characterized in that The ROI dynamic adjustment strategy also includes: When the shelf height changes by more than ±10%, the inter-layer matching algorithm based on SIFT feature points is triggered: Extract SIFT feature descriptors of the current layer and the reference layer; Eliminate mismatched points through the RANSAC algorithm; Calculate the affine transformation matrix based on the matching point pairs and update the ROI coordinates.

9. A high-value commodity off-shelf status tracking system based on computer vision, characterized in that: The device comprises: Coordinate library construction module: used to build a three-dimensional spatial coordinate library for the initial position of high-value goods, and establish a binding relationship between the uniqueness of high-value goods and their three-dimensional spatial coordinates; Motion detection module: This module detects the hand movements of people entering high-value areas. When the hand movement trajectory forms a spatial coupling with the displacement of goods, the person's physical features are extracted and marked as a person to be tracked. Product marking module: used to mark high-value products that are not returned within the first threshold time as products to be tracked, and establish a binding relationship between the products to be tracked and the people to be tracked; Trajectory prediction module: used to predict the movement trajectory of the person or product to be tracked. When the trajectory enters a high-risk area, the cross-camera tracking module is activated; Early warning module: used to send an early warning message when the payment of the tracked product has not been completed and the time it enters the high-risk area or leaves the high-value area exceeds the second threshold.

10. An electronic device, characterized in that: include: memory, processors and computer programs; The computer program is stored in the memory and configured to be executed by the processor to implement the method according to any one of claims 1 to 8.

Images