An abnormal taking behavior detection method of a vending machine

Abstract

The present application relates to the technical field of image processing, in particular to an abnormal taking behavior detection method of a self-service vending cabinet, comprising the following steps: step one: taking the user who opens the cabinet to take goods as a taking user, obtaining a taking identification video and intercepting a plurality of key frame images, and analyzing the key frame images to determine a plurality of shielding sensitive areas corresponding to the taking user; step two: constructing a sensitive area data set for the taking user based on the demarcated shielding sensitive areas, wherein the sensitive area data set comprises a plurality of skeleton coordinate combinations and a personal belongings feature vector; step three: triggering equal weight replacement detection when the weight of the sold goods and the returned goods is consistent; and step four: taking the sold goods and the returned goods corresponding to the triggered equal weight replacement detection as equal weight goods, and when the image similarity of the equal weight goods is greater than or equal to a preset similarity threshold, judging whether the equal weight goods disappear in the shielding sensitive area based on the taking and placing track and the sensitive area data set, and generating an abnormal replacement signal.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically to a method for detecting abnormal product retrieval behavior in unmanned vending machines. Background Technology

[0002] Unmanned vending machines, as a type of retail equipment based on automatic identification, automatic checkout, and remote monitoring, have been widely used in convenience stores, office areas, public places, and other scenarios. Users can independently take and put away goods after opening the machine. The system typically uses weight sensors, video surveillance equipment, or image recognition algorithms to collect product status data and complete transaction settlement. With the increasing application scenarios of unmanned vending machines, intelligent identification technologies for product removal, return, verification, and abnormal behavior detection within the machines have gradually developed, especially in product identification and detection, forming a relatively mature basic technology system.

[0003] However, existing vending machines often struggle to accurately identify whether items have been replaced when faced with goods of equal weight, similar packaging, or similar appearance, based solely on weight changes or differences in a single image frame. This is especially true when the user's hands, torso, or personal belongings obstruct the view of the goods, making it even more difficult to continuously track the true whereabouts and replacement process. The main reason for this is that items of equal weight rarely show significant differences in weighing results, making it easy for traditional detection methods based on single weight or single visual perception to miss detections. This leads to abnormal product retrieval behavior, particularly the difficulty in detecting and recording items replaced by other products of equal weight, thus presenting certain shortcomings. Summary of the Invention

[0004] To address the aforementioned shortcomings of existing technologies, this invention provides a method for detecting abnormal product retrieval behavior in unmanned vending machines, which can effectively solve the problem of difficulty in detecting repeated replacement behavior in existing technologies.

[0005] To achieve the above objectives, the present invention provides the following technical solution: This invention provides a method for detecting abnormal product retrieval behavior in unmanned vending machines, comprising at least the following steps: Step 1: Record the user who opens the cabinet to pick up the goods as the pickup user, acquire the pickup recognition video and extract multiple keyframe images, analyze the keyframe images to determine multiple occlusion-sensitive areas corresponding to the pickup user, including: The preprocessed keyframe images are filtered to obtain valid images. Based on the valid images, the human body contour of the picking user is determined and a skeleton coordinate system is constructed. The corresponding complete body contour, personal belongings contour and clothing coverage area are identified, and the range of multiple limb contours in the complete body contour is determined. Based on the location of the limb outline and the appearance of the clothing outline, we analyze whether each limb outline has the conditions to accommodate goods, and the area that has the conditions to accommodate goods is recorded as the occlusion sensitive area. Step 2: Based on the defined occlusion sensitive areas, construct a sensitive area dataset for the picking user. The sensitive area dataset includes multiple skeleton coordinate combinations and feature vectors of personal items. Step 3: Based on the pickup video, identify the pickup and return behaviors, as well as the corresponding sold and returned goods. When the sold and returned goods have the same weight, trigger the equal weight replacement detection. Step 4: Record the sold and returned goods that trigger the equal weight replacement detection as equal weight goods, collect images of equal weight goods for similarity comparison, and generate an abnormal replacement signal when the similarity is less than the preset similarity threshold. When the similarity is greater than or equal to the preset similarity threshold, the picking and placing trajectory of goods of equal weight is identified. Based on the picking and placing trajectory and the sensitive area dataset, it is determined whether the goods of equal weight disappear in the occluded sensitive area and an abnormal replacement signal is generated.

[0006] Furthermore, the process for determining the occluded sensitive area is as follows: S1: Perform time-series alignment on multiple keyframe images, filter out blurry frames and severely occluded frames in the keyframe images, perform noise reduction, distortion correction, brightness equalization and resolution unification on the keyframe images, compare multiple keyframe images, select the primary keyframe image with clear and complete user outline as the effective image, and identify the human outline in the effective image as the target outline. Based on the human pose estimation model, the positions of multiple skeleton key points of the target contour are marked. The skeleton coordinate system of the picking user is established by connecting multiple skeleton key points. The skeleton coordinate system includes multiple skeleton center lines. The complete body contour and the corresponding multiple limb contour ranges are determined by combining the skeleton coordinate system. The limb contour ranges include the upper body trunk region, the lower body trunk region, the two arm regions and the two leg regions. S2: The target contour area that is not a complete body outline is recorded as the accessory outline, and the accessory outline is recorded as the occluded sensitive area. S3: Identify the clothing-covered area within the limb outline. Multiple torso-accommodating areas are preset. For clothing-covered areas used to store items, the overlapping part of the clothing-covered area and the torso-accommodating area is recorded as the occlusion sensitive area. S4: Record the non-occluded sensitive area in the clothing-covered area as the independent identification part. Based on the skeleton coordinate system and the limb contours adjacent to the independent identification part, analyze the contour features of the independent identification part to determine whether there are abnormal raised areas in the independent identification part. Record the abnormal raised areas of the independent identification part as occlusion sensitive areas.

[0007] Furthermore, the effective image selection process is as follows: The human body contour in each keyframe image is identified and recorded as the identified contour. The contour recognition parameters are calculated based on the total area of ​​the identified contour and the area of ​​the head contour in the identified contour. The keyframe image with the largest contour recognition parameters is recorded as the valid image.

[0008] Furthermore, the process for determining abnormally raised areas is as follows: Obtain the outer contour curve of the independently identified part, construct the distance distribution curve of the outer contour curve relative to the center line of the skeleton, and record the independently identified parts that are symmetrical on both sides of the body as symmetrical part combinations. Compare and analyze the distance distribution curve of each symmetrical part combination, calculate the symmetry anomaly value and width anomaly value of each independently identified part, and set corresponding anomaly thresholds. When either the symmetry anomaly value or the width anomaly value is greater than or equal to the corresponding anomaly threshold, the corresponding contour area is recorded as an abnormal bulge area.

[0009] Furthermore, the calculation process for independently identifying partially symmetric outliers and width outliers is as follows: S41: The distance distribution curve corresponding to the independently identified part is recorded as the target distribution curve. The target distribution curve is smoothed using a preset smoothing interval. Multiple uniformly distributed sampling points are set on the target distribution curve. A fitted straight line is constructed based on the coordinates of the sampling points and recorded as the reference straight line. S42: The portion of the target distribution curve that lies above the reference line is recorded as the protruding part. The number of sampling points corresponding to the protruding part is recorded as the protruding number. A threshold for the protruding number is preset. Protruding parts with a protruding number greater than the threshold are marked as abnormal protruding parts. A distance distribution curve symmetrical to the target distribution curve is recorded as the reference distribution curve. Abnormal protruding parts corresponding to the reference distribution curve are recorded as reference protruding parts. The symmetrical outlier is calculated based on the overlap between the reference protruding part and the abnormal protruding part. The smaller the overlap, the larger the symmetrical outlier. S43: When the independently identified part corresponds to two target distribution curves, calculate the relative distance between the two target distribution curves and record it as the width value, and obtain the distribution curve of the width value along the center line of the skeleton and record it as the width distribution curve; S44: Construct a fitted straight line based on multiple sampling points on the width distribution curve, denoted as the width straight line. The part of the width distribution curve located above the width straight line is denoted as the width abnormal part. Remove the width abnormal parts whose length is less than the preset length threshold. Calculate the width abnormal value based on the average offset of the width value of the width abnormal part relative to the width straight line. Set the width anomaly value of the independent identification portion of a target distribution curve with only one target distribution curve to 0.

[0010] Furthermore, the process of obtaining the skeleton coordinate combination is as follows: Bind the occluded sensitive area to its corresponding skeleton centerline, and assign a positioning coordinate to each skeleton centerline with the head vertex in the skeleton coordinate system as the reference point. The corresponding one-dimensional coordinate axis is determined based on the center line of the skeleton, with the positive direction of the coordinate axis pointing away from the reference point. The node coordinates are determined based on the distribution range of the occlusion sensitive area on the corresponding center line of the skeleton. The skeleton coordinate combination is obtained by combining the positioning coordinates and the node coordinates.

[0011] Furthermore, distortion correction is performed on the node coordinates to compensate for the differences in unit quantities of the skeleton coordinate system caused by different shooting distances. The correction process is as follows: P1: A ground reference coordinate system is preset. The key point of the toe in the skeleton coordinate system is recorded as the positioning anchor point. Based on the coordinates of the positioning anchor point in the ground reference coordinate system, the positioning coordinates are recorded. A three-dimensional spatial coordinate system is constructed. The skeleton center line corresponding to the lower leg is represented in the three-dimensional spatial coordinate system and recorded as the anchoring center line. The anchoring center line is perpendicular to the ground reference coordinate system. A plane perpendicular to the horizontal component of the camera shooting direction is drawn through the anchoring center line and recorded as the anchoring plane. P2: A preset image pixel space projection model is used to determine the imaging ray corresponding to each pixel in each valid image; Based on the endpoint positions of each skeleton centerline in the effective image, the endpoint rays of each skeleton centerline are determined. The intersection of the endpoint rays and the anchoring plane is recorded as the spatial projection point of the skeleton centerline endpoint, thus obtaining the actual expression of the skeleton coordinate system in the three-dimensional spatial coordinate system, and representing the coordinates of each node in the three-dimensional coordinate system. P3: Select a skeleton centerline as the basic reference length to establish skeleton proportional constraints. Based on the symmetry of the skeleton centerlines in the skeleton coordinate system, correct the lengths of abnormally long skeleton centerlines while maintaining their direction. P4: The skeleton centerline that needs to be length corrected in P3 is called the correction centerline. Based on the proportional coordinates of the node coordinates on the correction centerline, the corresponding coordinate positions are marked on the skeleton centerline after length correction. Using the skeleton coordinate system in the anchoring plane as the reference coordinate system, obtain the coordinates of each node and regenerate the skeleton coordinate combination.

[0012] Furthermore, the revised conditions are as follows: Rule 1: Make the center lines of the two symmetrical skeletons have the same length, and the length is equal to the maximum of the two lengths; Rule 2: Calculate the lower limit of the length corresponding to each skeleton centerline based on the basic reference length. When the length of any skeleton centerline is less than the lower limit, set its length to be equal to the lower limit.

[0013] Furthermore, the disappearance judgment process is as follows: The times when the weight sensor values ​​change corresponding to the removal and return actions are recorded as the removal time and the return time, respectively, and the time intervals corresponding to the removal time and the return time are recorded as the replacement interval. The unit time interval is obtained by dividing the minimum width of the goods of equal weight by the preset speed reference value. Multiple picking images in the replacement interval are extracted using the unit time interval as the extraction interval. The goods of equal weight in the picking images are identified. Picking images with goods of equal weight are recorded as existing images and assigned a value of 1. Picking images without goods of equal weight are recorded as missing images and assigned a value of 0. Constructing a time series of pickup images , This indicates the assignment of values ​​to the pickup image with sequence number i, where i = 1, 2, ..., j. If there is a partial sequence in the pickup image time series... A sequence is given that m > Z and all elements in the sequence are 0. , If the starting and ending elements of a partial sequence are respectively, then extract... Multiple adjacent existing images are denoted as target images, where Z is a preset quantity threshold; Based on the decreasing contour variation pattern of equally heavy objects in the target image, a disappearance trajectory of equally heavy objects is constructed. When the disappearance trajectory determines the disappearance area corresponding to the equally heavy object, if the disappearance area belongs to the occlusion sensitive area, an abnormal replacement signal is generated.

[0014] Furthermore, the process for determining the disappearance region is as follows: The outlines of objects of equal weight in each target image are recorded as visual outlines, and the completeness of each visual outline is determined by combining the complete outlines of the objects of equal weight. Visual contours with a completeness lower than a preset completeness threshold are denoted as partial contours. Based on the prior knowledge of the complete contours of medium-weight objects in the image, contour extraction, missing segment localization, and shape prior completion are performed on the partial contours. The complete mask of the corresponding object is output, and the complete mask area corresponding to the partial contour with the lowest completeness is denoted as the disappearing area.

[0015] The technical solution provided by this invention has the following advantages compared with the known prior art: 1. By filtering keyframes in the pickup video, constructing a human skeleton coordinate system, and recognizing the contours of the body and personal belongings, and further combining clothing coverage features, symmetry anomalies, width anomalies, and abnormal bulge features to determine occlusion-sensitive areas, standardized localization of parts of the user's body surface and personal belongings that may contain similar goods is achieved. Compared with existing technologies that rely solely on single-frame human appearance or simple part selection, this solution can uniformly map image information that fluctuates with changes in posture, shooting angle, and individual body shape to the skeleton coordinate system, avoiding misjudgments caused by perspective distortion, contour occlusion, and clothing wrinkles. It solves the problems of unstable extraction of occluded sensitive areas and difficulty in cross-temporal comparison, improving the accuracy and generalization ability of sensitive area recognition.

[0016] 2. By further comparing image similarity, analyzing pick-up and drop-off trajectories, determining the disappearance of occluded sensitive areas, and locating disappearance areas using contour completion for sold and returned goods of the same weight, a multi-dimensional joint detection of equal-weight replacement behavior is achieved. Compared to existing technologies that rely solely on weight changes or differences in product appearance, this solution can still detect abnormal transfer processes by linking temporal trajectories and spatial regions, even when goods are of the same weight, similar in appearance, and the pick-up and drop-off actions are brief and concealed. It is particularly suitable for identifying abnormal pick-up behaviors that are highly concealed, short in process, and difficult to record evidence, thus improving the stability, continuity, and traceability of the detection results. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0018] Figure 1 This is an overall module block diagram of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0020] The present invention will be further described below with reference to embodiments.

[0021] See Figure 1 A method for detecting abnormal product retrieval behavior in unmanned vending machines is provided to identify abnormal substitution behavior during the product retrieval process, especially substitution of equal weight. The method includes the following steps: Step 1: When a user enters the designated area in front of the vending machine, the video stream data collected by the monitoring equipment (usually set on top of the vending machine to capture video in front of the vending machine, or it can be set up separately at a certain distance from the vending machine) is recorded as the product retrieval identification video. If the user opens the vending machine to retrieve an item, the product retrieval identification video is retained and analyzed. Multiple key frame images are extracted and recorded as the front recognition images. The user who opens the vending machine to retrieve an item is recorded as the retrieving user. Based on the front recognition images, multiple occlusion sensitive areas corresponding to the retrieving user are determined. The occlusion sensitive areas refer to the areas where the retrieving user may have stored similar items (i.e., objects similar in weight and appearance to the items in the vending machine). If the user does not open the vending machine to retrieve an item, the product retrieval identification video is deleted after the user leaves the designated area.

[0022] It should be noted that by continuously recording videos of users entering designated areas and deciding whether to retain them based on whether subsequent pickup behavior is triggered, it is possible to retain pickup identification videos of all pickup users as much as possible, while avoiding interference from non-pickup users. On this basis, identifying occluded sensitive areas in the pickup identification videos can be used for the verification process of abnormal pickup behavior in the subsequent pickup process, enabling targeted verification for different users.

[0023] Specifically, the process for determining the sensitive areas to be obscured is as follows: S1: Perform temporal alignment on multiple keyframe images, filter out blurry frames and severely occluded frames, and perform denoising, distortion correction, brightness equalization and resolution unification on the keyframe images to improve the stability of subsequent human detection and key part recognition. Compare multiple keyframe images and select the primary keyframe image with clear and complete user contours as the effective image. Identify the human contour in the effective image and record it as the target contour (objects that are continuous with the user's torso in the image are considered as part of the target contour, such as the user's handbag, shoulder bag and backpack and other personal items).

[0024] Recognizing human body contours in images is an existing technology and will not be discussed further here.

[0025] Based on the human pose estimation model (which belongs to the deep learning model in the existing technology), the positions of multiple skeleton key points of the target contour corresponding to the picking user are marked. The skeleton key points include the shoulders, elbows, chest, waist, hips, etc. The skeleton coordinate system of the picking user is established by connecting multiple skeleton key points. The skeleton coordinate system includes multiple skeleton center lines (line segments). Then, based on the spatial relationship of the skeleton key points, the complete body contour (i.e., only the contour area of ​​the human body) and multiple limb contour ranges are determined. The limb contour ranges include the upper body trunk area, the lower body trunk area, the two arm areas, and the two leg areas. It should be noted that the skeleton coordinate system can transform the pixel coordinates in the original image that vary with viewpoint, posture, and body shape into a normalized space that is referenced to the human skeletal structure. This coordinate system can significantly reduce the geometric distortion caused by individual differences such as the user's distance from the camera, standing direction, height, and weight. This makes the semantic definitions of "upper torso" and "arm area" no longer dependent on absolute image positions, but based on the relative spatial relationships of skeleton key points for a consistent description, thereby significantly improving the generalization ability and accuracy of abnormal pickup behavior detection.

[0026] S2: The target contour area that is not a complete body outline is recorded as the personal item outline, and the personal item outline is recorded as the occlusion sensitive area (i.e. the outline corresponding to the carried item). For users picking up goods carrying personal items, the items they carry should be directly regarded as the occlusion sensitive area, and these items have the conditions to accommodate similar goods.

[0027] The effective image selection process is as follows: The human body contour in each keyframe image is identified as the identified contour. The contour recognition parameter is calculated based on the total area of ​​the identified contour and the area of ​​the head contour in the identified contour. The larger the contour recognition parameter, the more complete and clear the contour in the keyframe image is. The keyframe image with the largest contour recognition parameter is recorded as the valid image.

[0028] Contour recognition parameters The calculation formula is ;in Represents the total area of ​​the identified contour. Represents the area of ​​the head outline. To preset the head area baseline value, , All of these are preset weighting coefficients (set by staff based on experience).

[0029] It should be noted that the first term in the contour recognition parameter calculation formula is used to evaluate the completeness of the recognized contour. This is achieved by comparing the complete contour (which typically changes with the movement of the user picking up the goods) with the head contour (which usually remains constant), thus assessing the completeness of the complete contour in each valid image. The second term is used to evaluate the sharpness of the recognized contour. Generally, a larger head contour indicates that the user is getting closer to the camera, making the human contour clearer in the surveillance image. By calculating the contour recognition parameters for filtering valid images, relatively clear and complete images can be selected from multiple keyframe images to identify key feature data in the human contour, thereby improving the accuracy of the recognition process.

[0030] S3: Further identify whether there is clothing covering the limb outline range, and extract the clothing covering area in the limb outline range. There are multiple torso accommodating areas. The torso accommodating areas correspond to the clothing areas that the human body usually uses to store items (defined by the staff). In a specific embodiment, the torso accommodating areas include the chest and abdomen, waist, hips and thighs. The overlapping part of the clothing covering area and the torso accommodating area is recorded as the occluded sensitive area. It should be noted that the overlapping area between the clothing-covered area and the torso-containing area corresponds to the area on the user's body that has the conditions to accommodate similar goods. This area should also be regarded as a sensitive area that is obscured, and used for the determination of abnormal pickup behavior in the subsequent process.

[0031] S4: Record the non-occluded sensitive areas within the clothing-covered area as independent identification parts (e.g., sleeves, shoulders, and lower legs). Based on the skeletal coordinate system and the limb contours adjacent to the independent identification parts (limb contours in the non-clothing-covered areas), analyze the contour features of the independent identification parts to determine whether there are abnormal raised areas in the independent identification parts. Record the abnormal raised areas of the independent identification parts as occlusion sensitive areas.

[0032] It should be noted that the independently identifiable area, which is the area covered by clothing that is not usually used for storing objects on the human body, can be used to store similar goods in some special cases. Therefore, when it meets the abnormal bulge characteristics, it is also recorded as an occluded sensitive area.

[0033] By defining occlusion-sensitive areas, the detection of abnormal behavior is no longer limited to "determining whether there is occlusion in the product image," but further determines "whether the occlusion occurs in an area that may carry replacement behavior." This distinguishes a large number of irrelevant occlusion factors, such as human bodies, personal belongings, and clothing folds, from the real abnormal replacement paths. Consequently, only occlusions falling within high-risk areas are included in the anomaly analysis, which can significantly reduce the interference caused by invalid occlusions.

[0034] More specifically, the determination process of the abnormal bulge area is as follows: Obtain the outer contour curve of the independently recognized part (that is, the partial contour belonging to the complete body contour, excluding the dividing line with other contours. When the independently recognized part is the arm part or the calf part, it includes two outer contour curves on both sides of the skeleton center line), construct the distance distribution curve of the outer contour curve relative to the skeleton center line, record the independently recognized parts symmetric on both sides of the body as a symmetric part combination, conduct a comparative analysis on the distance distribution curves of each symmetric part combination (each distance distribution curve can find a corresponding symmetric curve on the other side of the body), calculate the symmetric abnormality value and width abnormality value of each independently recognized part, which are used to evaluate whether there is symmetric abnormality (significantly not meeting the symmetry condition) or width abnormality (significantly wider than the normal clothing contour distribution) in the outer contour curve. Corresponding abnormality thresholds are set respectively. When any one of the symmetric abnormality value and width abnormality value is greater than or equal to the corresponding abnormality threshold, the corresponding contour area is recorded as the abnormal bulge area.

[0035] It should be noted that the skeleton center line corresponds to the abstract simulated line segment of the bones of each part of the human body (such as the line segment corresponding to the upper edge of the scapula). That is to say, the skeleton coordinate system is composed of multiple line segments (skeleton center lines) connected. The connection points of adjacent line segments usually correspond to joints. The whole skeleton coordinate system looks like a shape structure formed by two "V" characters stacked up and down.

[0036] Furthermore, the calculation process of the symmetric abnormality value and width abnormality value of the independently recognized part is as follows: S41: Record the distance distribution curve corresponding to the independently recognized part as the target distribution curve, and perform smoothing processing on the target distribution curve using a preset smoothing interval (that is, divide the target distribution curve into multiple unit curves according to the smoothing interval, and then perform smoothing processing on each unit curve to fade the local curve mutation and reduce the influence of wrinkles), so that the target distribution curve is converted into a smooth and continuous curve within each unit smoothing interval. Set multiple uniformly distributed sampling points on the target distribution curve, and construct a fitting straight line based on the coordinates of the sampling points as the reference straight line; S42: The portion of the target distribution curve that lies above the reference line is recorded as the protruding part. The number of sampling points corresponding to the protruding part is recorded as the protruding number. A threshold for the protruding number is preset. Protruding parts with a protruding number greater than the threshold are marked as abnormal protruding parts. A distance distribution curve symmetrical to the target distribution curve is recorded as the reference distribution curve. Abnormal protruding parts corresponding to the reference distribution curve are recorded as reference protruding parts. The symmetrical anomaly value is calculated based on the overlap between the reference protruding part and the abnormal protruding part (i.e., the overlap of their distribution ranges on the center line of the skeleton). The smaller the overlap, the larger the symmetrical anomaly value (abnormal protruding parts with symmetrical anomaly values ​​greater than the corresponding threshold are abnormal bulges, and the corresponding occlusion sensitive areas are determined based on these parts). S43: When the independently identified part corresponds to two target distribution curves, calculate the relative distance between the two target distribution curves and record it as the width value (with the skeleton center line as the baseline, calculate the sum of the distances from the same point to the target distribution curves on both sides to obtain the width value), and obtain the distribution curve of the width value along the skeleton center line and record it as the width distribution curve, which reflects the change of the width of the clothing outline of the independently identified part; S44: Construct a fitted straight line based on multiple sampling points on the width distribution curve, denoted as the width line. The portion of the width distribution curve above the width line is denoted as the width anomaly. Remove width anomalies whose length is less than a preset length threshold (to reduce local wrinkle interference). Calculate the width anomaly value based on the average offset of the width value of the width anomaly relative to the width line (width anomalies whose width anomalies are greater than the corresponding threshold are abnormal bulges, and the corresponding occlusion sensitive area is determined based on this part). When an independently identified portion corresponds to a target distribution curve, its width anomaly value is set to 0.

[0037] It should be noted that the width outlier is used to reflect whether there is a part in the area covered by the clothing of the person picking up the goods that is significantly wider than the adjacent area, and can preliminarily determine the difference in width between this part and the normal width. If there are hidden similar goods in the area covered by the clothing, the difference usually corresponds to the width of the similar goods.

[0038] It is worth noting that by smoothing, fitting, and symmetrically comparing the clothing outline distribution of independently identified areas by the user, the degree of abnormal bulging in the clothing-covered area can be quantitatively analyzed from three dimensions: local protrusion, symmetry anomaly, and width anomaly. Compared to methods that rely solely on single-frame appearance judgment or simple occlusion detection, this scheme cleverly utilizes the linear and symmetrical features of the clothing outline and can effectively reduce interference from wrinkles, pose changes, and local noise, improving the accuracy and stability of abnormal bulging area identification. This provides a reliable basis for subsequent judgment of abnormal retrieval behaviors such as equal weight replacement and concealment.

[0039] Step 2: Based on the occlusion-sensitive areas defined in S1-S3, construct a sensitive area dataset for the picking user. The sensitive area dataset includes multiple skeleton coordinate combinations and feature vectors of personal items. Thus, for any image captured subsequently containing the outline of the picking user's body, quickly determine whether any point in the subsequent image belongs to the occlusion-sensitive area based on the sensitive area dataset, thereby using it for occlusion interference judgment between the goods and the occlusion-sensitive areas.

[0040] Specifically, the process of obtaining the skeleton coordinate combination is as follows: The occlusion-sensitive areas are bound to their corresponding skeletal centerlines. Each skeletal centerline is assigned a positioning coordinate, with the head vertex in the skeletal coordinate system as the reference point. The positioning coordinate is used to determine the relative position of each skeletal centerline in the skeletal coordinate system (the human skeleton is distributed according to a fixed pattern, so it can be determined by one-dimensional positioning coordinates, such as a serial number), thereby achieving the positioning of each skeletal centerline. Based on the skeletal centerline, the corresponding one-dimensional coordinate axis is determined, with the positive direction of the coordinate axis pointing away from the reference point. The node coordinates (expressed as interval endpoints) are determined based on the distribution range of the occlusion-sensitive areas on the corresponding skeletal centerlines. The positioning coordinates and node coordinates are combined to obtain the skeletal coordinate combination.

[0041] It should be noted that skeleton coordinate combinations are used to locate each occlusion-sensitive region in the skeleton coordinate system. By binding the occlusion-sensitive region to the skeleton centerline and combining it with the head vertex reference point, positioning coordinates, and node coordinates to construct skeleton coordinate combinations, the originally scattered and posture-sensitive region information can be uniformly mapped to a standardized human skeleton coordinate system, thereby accurately representing the relative position and distribution range of each sensitive region in the human body. This not only facilitates continuous tracking and consistency comparison of the same sensitive region in images taken at different times, but also improves the stability and reusability of occlusion region recognition, enabling the detection of weighted replacement behaviors and reducing positioning errors caused by changes in human movement.

[0042] More specifically, distortion correction is performed on the node coordinates to compensate for the differences in unit quantities of the skeleton coordinate system caused by different shooting distances. The correction process is as follows: P1: A ground reference coordinate system is preset. The key point of the toe in the skeleton coordinate system is recorded as the positioning anchor point. Based on the coordinates of the positioning anchor point in the ground reference coordinate system, the positioning coordinates are recorded. A three-dimensional spatial coordinate system is constructed. The skeleton center line corresponding to the lower leg is represented in the three-dimensional spatial coordinate system and recorded as the anchoring center line. The anchoring center line is perpendicular to the ground reference coordinate system. A plane perpendicular to the horizontal component of the camera shooting direction is drawn through the anchoring center line and recorded as the anchoring plane. It should be noted that using the ground as a reference to determine the standing position of the person picking up the goods, and thus the coordinate position of the person picking up the goods in the three-dimensional coordinate system, can avoid the interference of individual differences among different people picking up goods and obtain a relatively accurate positioning result.

[0043] P2: A preset image pixel spatial projection model (obtained based on the spatial position and parameters of the camera) is used to determine the imaging ray (multiple rays emanating from the camera position pointing towards the ground) corresponding to each pixel in each effective image. Based on the image pixel spatial projection model and combined with the endpoint positions of each skeleton centerline in the effective image, the endpoint rays of each skeleton centerline are determined. The intersection of the endpoint rays and the anchoring plane is recorded as the spatial projection point of the skeleton centerline endpoint, thus obtaining the actual expression of the skeleton coordinate system in the three-dimensional spatial coordinate system, and representing the coordinates of each node in the three-dimensional coordinate system. P3: Select a skeleton centerline (in one specific embodiment, select the skeleton centerline corresponding to the lower leg bone, because it is usually perpendicular to the ground, and the calculated length is closer to the actual length) as the basic reference length to establish skeleton proportional constraints. Combining the symmetry of each skeleton centerline in the skeleton coordinate system, correct the length of skeleton centerlines with abnormal lengths, while keeping the direction unchanged. The correction conditions are as follows: Rule 1: Make the center lines of the two symmetrical skeletons have the same length, and the length is equal to the maximum of the two lengths; Rule 2: Calculate the lower limit of the length corresponding to each skeleton centerline based on the basic reference length. When the length of any skeleton centerline is less than the lower limit, set its length to be equal to the lower limit.

[0044] P4: The skeleton centerline that needs to be length corrected in P3 is called the correction centerline. Based on the proportional coordinates of the node coordinates on the correction centerline, the corresponding coordinate positions are marked on the skeleton centerline after length correction. Using the skeleton coordinate system in the anchoring plane (which has been corrected by P1-P4) as the reference coordinate system, obtain the coordinates of each node and regenerate the skeleton coordinate combination.

[0045] It should be noted that by correcting the skeleton coordinate system through the above steps, the comparability of images taken at different times can be further optimized. In particular, after transforming the skeleton coordinate system to a two-dimensional coordinate system in real space, the scale differences caused by different shooting times, different standing distances, and different human posture changes can be significantly reduced. Especially in the skeleton coordinate combination output stage of the occluded sensitive area, the occluded sensitive area is projected onto the real space reference coordinates for equal weight replacement detection. Thus, images taken at different times can use the skeleton coordinate combination of the occluded sensitive area as a reference to determine whether the product has entered the occluded sensitive area.

[0046] It is worth noting that if the modified skeleton coordinate combination is used, when determining whether the product interferes with the occlusion sensitive area, the occlusion interference analysis should also be performed on the anchor plane (instead of inferring the occlusion sensitive area on the user's human body contour through the skeleton coordinate combination; that is, the occlusion sensitive area is only determined once in the valid image, and is used as a reference object to determine whether the product enters the occlusion sensitive area), so as to ensure that the two have spatial consistency.

[0047] Specifically, the process of obtaining the feature vector of personal items is as follows: The occlusion-sensitive area corresponding to the non-complete outline of the body is denoted as the body object region. Image features are extracted for the body object region and converted into feature vectors that can be used to compare and judge similarity. Then, based on the feature vectors, the image region that is similar to it in any subsequent image is determined.

[0048] Step 3: Obtain the video corresponding to the user's opening and picking up of goods and record it as the picking up video. Based on the preset image recognition model and behavior analysis model combined with the numerical changes of the weight sensor, identify the picking behavior in the picking up video and divide the picking behavior into taking out behavior and putting back behavior. Taking out behavior and putting back behavior correspond to sold goods and returned goods, respectively (the image recognition model and behavior analysis model are existing technologies that can combine video recognition and weight sensing to determine the taking out of goods, putting back goods or inventory changes, which will not be elaborated on here). When the weight of sold goods and returned goods are the same, trigger equal weight replacement detection. Step 4: Record the sold and returned goods that trigger the equal weight replacement detection as equal weight goods, collect images of equal weight goods for similarity comparison, and generate an abnormal replacement signal when the similarity is less than the preset similarity threshold. When the similarity is greater than or equal to the preset similarity threshold, the picking and placing trajectory of equal-weight goods is identified based on the picking video. Based on the picking and placing trajectory and the sensitive area dataset, it is determined whether the equal-weight goods disappear in the occluded sensitive area. If they disappear and the disappearance duration is greater than or equal to the preset duration threshold, an abnormal replacement signal is generated.

[0049] Specifically, the disappearance determination process is as follows: The times when the weight sensor values ​​change during the removal and return actions are recorded as the removal time and the return time, respectively (the value decreases when removed and increases when returned). The time intervals corresponding to the removal and return times are recorded as the replacement intervals. The minimum width of the goods of equal weight is divided by a preset speed reference value (corresponding to the upper limit of the speed of picking up the items, which is 1m / s in a specific embodiment) to obtain the unit time interval. Multiple picking images within the replacement interval are extracted using the unit time interval as the extraction interval. The goods of equal weight in the picking images are identified based on the image recognition algorithm (identifying a known object in the image is the prior art). Picking images with goods of equal weight are recorded as existing images and assigned a value of 1, while picking images without goods of equal weight are recorded as missing images and assigned a value of 0. Constructing a time series of pickup images , This indicates the assignment of values ​​to the pickup image with sequence number i, where i = 1, 2, ..., j. If there is a partial sequence in the pickup image time series... A sequence is given that m > Z and all elements in the sequence are 0. , If the starting and ending elements of a partial sequence are respectively, then extract... Multiple adjacent existing images are denoted as target images, where Z is a preset quantity threshold; Based on the decreasing contour variation of equally heavy objects in the target image, a disappearance trajectory of equally heavy objects is constructed. When the disappearance trajectory determines the disappearance area corresponding to the equally heavy object, if the disappearance area belongs to the occlusion sensitive area (when the disappearance area coincides with the human contour, the human contour area corresponding to the disappearance area is transformed to the anchor point plane, and then checked with the corresponding skeleton coordinate combination in the skeleton coordinate system; when the disappearance area does not coincide with the human contour, it is directly checked with the occlusion sensitive area corresponding to the contour of the personal object), then an abnormal replacement signal is generated.

[0050] It should be noted that, under the premise of consistent weight, by fusing image similarity and temporal trajectory analysis during the picking and placing process, and further introducing constraints on occlusion-sensitive areas, the system can determine whether abnormal replacement of goods has occurred. In particular, by identifying the continuous "disappearance" of goods within key time intervals and combining this with whether the location of the disappearance belongs to a high-risk occlusion area, the system characterizes abnormal behavior from two dimensions: temporal continuity and spatial correlation. This effectively improves the accuracy and stability of detecting highly concealed equal-weight replacement behavior.

[0051] More specifically, the process for determining the disappearance region is as follows: The outlines of objects of equal weight in each target image are recorded as visual outlines, and the completeness of each visual outline is determined by combining the complete outlines of the objects of equal weight. Visual contours with a completeness lower than a preset completeness threshold are denoted as partial contours. Based on the prior knowledge of the complete contours of medium-weight objects in the image, contour extraction, missing segment localization, and shape prior completion are performed on the partial contours to output the complete mask of the corresponding object, so as to realize the structural restoration and subsequent recognition of the occluded target. The complete mask area corresponding to the partial contour with the lowest completeness is denoted as the disappearance area.

[0052] It should be noted that the method of "completing the occluded part when the complete outline is known" is existing technology and will not be elaborated on here. By evaluating the completeness of the visible outline of an object of equal weight in each target image, and using the complete outline prior to complete the occluded structure when the outline is severely missing, the target that was originally only partially visible can be restored to a more realistic complete target representation. This allows for more accurate location of the object's disappearance area in the image, providing a more stable spatial basis for subsequent judgment of whether the product has been moved to an occluded sensitive area, and reducing misjudgments caused by single-frame occlusion, viewpoint changes, or local missing parts.

[0053] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions will not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting abnormal product retrieval behavior in unmanned vending machines, characterized in that, Includes the following steps: Step 1: Record the user who opens the cabinet to pick up the goods as the pickup user, acquire the pickup recognition video and extract multiple keyframe images, analyze the keyframe images to determine multiple occlusion-sensitive areas corresponding to the pickup user, including: The preprocessed keyframe images are filtered to obtain valid images. Based on the valid images, the human body contour of the picking user is determined and a skeleton coordinate system is constructed. The corresponding complete body contour, personal belongings contour and clothing coverage area are identified, and the range of multiple limb contours in the complete body contour is determined. Based on the location of the limb outline and the appearance of the clothing outline, we analyze whether each limb outline has the conditions to accommodate goods, and the area that has the conditions to accommodate goods is recorded as the occlusion sensitive area. Step 2: Based on the defined occlusion sensitive areas, construct a sensitive area dataset for the picking user. The sensitive area dataset includes multiple skeleton coordinate combinations and feature vectors of personal items. Step 3: Based on the pickup video, identify the pickup and return behaviors, as well as the corresponding sold and returned goods. When the sold and returned goods have the same weight, trigger the equal weight replacement detection. Step 4: Record the sold and returned goods that trigger the equal weight replacement detection as equal weight goods, collect images of equal weight goods for similarity comparison, and generate an abnormal replacement signal when the similarity is less than the preset similarity threshold. When the similarity is greater than or equal to the preset similarity threshold, the picking and placing trajectory of goods of equal weight is identified. Based on the picking and placing trajectory and the sensitive area dataset, it is determined whether the goods of equal weight disappear in the occluded sensitive area and an abnormal replacement signal is generated.

2. The method for detecting abnormal product retrieval behavior of an unmanned vending machine according to claim 1, characterized in that, The process for determining the sensitive areas to be obscured is as follows: S1: Perform time-series alignment on multiple keyframe images, filter out blurry frames and severely occluded frames in the keyframe images, perform noise reduction, distortion correction, brightness equalization and resolution unification on the keyframe images, compare multiple keyframe images, select the primary keyframe image with clear and complete user outline as the effective image, and identify the human outline in the effective image as the target outline. Based on the human pose estimation model, the positions of multiple skeleton key points of the target contour are marked. The skeleton coordinate system of the picking user is established by connecting multiple skeleton key points. The skeleton coordinate system includes multiple skeleton center lines. The complete body contour and the corresponding multiple limb contour ranges are determined by combining the skeleton coordinate system. The limb contour ranges include the upper body trunk region, the lower body trunk region, the two arm regions and the two leg regions. S2: The target contour area that is not a complete body outline is recorded as the accessory outline, and the accessory outline is recorded as the occluded sensitive area. S3: Identify the clothing-covered area within the limb outline. Multiple torso-accommodating areas are preset. For clothing-covered areas used to store items, the overlapping part of the clothing-covered area and the torso-accommodating area is recorded as the occlusion sensitive area. S4: Record the non-occluded sensitive area in the clothing-covered area as the independent identification part. Based on the skeleton coordinate system and the limb contours adjacent to the independent identification part, analyze the contour features of the independent identification part to determine whether there are abnormal raised areas in the independent identification part. Record the abnormal raised areas of the independent identification part as occlusion sensitive areas.

3. The method for detecting abnormal product retrieval behavior of an unmanned vending machine according to claim 2, characterized in that, The valid image selection process is as follows: The human body contour in each keyframe image is identified and recorded as the identified contour. The contour recognition parameters are calculated based on the total area of ​​the identified contour and the area of ​​the head contour in the identified contour. The keyframe image with the largest contour recognition parameters is recorded as the valid image.

4. The method for detecting abnormal product retrieval behavior of an unmanned vending machine according to claim 3, characterized in that, The process for determining abnormally raised areas is as follows: Obtain the outer contour curve of the independently identified part, construct the distance distribution curve of the outer contour curve relative to the center line of the skeleton, and record the independently identified parts that are symmetrical on both sides of the body as symmetrical part combinations. Compare and analyze the distance distribution curve of each symmetrical part combination, calculate the symmetry anomaly value and width anomaly value of each independently identified part, and set corresponding anomaly thresholds. When either the symmetry anomaly value or the width anomaly value is greater than or equal to the corresponding anomaly threshold, the corresponding contour area is recorded as an abnormal bulge area.

5. The method for detecting abnormal product retrieval behavior of an unmanned vending machine according to claim 4, characterized in that, The calculation process for independently identifying partially symmetrical outliers and width outliers is as follows: S41: The distance distribution curve corresponding to the independently identified part is recorded as the target distribution curve. The target distribution curve is smoothed using a preset smoothing interval. Multiple uniformly distributed sampling points are set on the target distribution curve. A fitted straight line is constructed based on the coordinates of the sampling points and recorded as the reference straight line. S42: The portion of the target distribution curve that lies above the reference line is recorded as the protruding part. The number of sampling points corresponding to the protruding part is recorded as the protruding number. A threshold for the protruding number is preset. Protruding parts with a protruding number greater than the threshold are marked as abnormal protruding parts. A distance distribution curve symmetrical to the target distribution curve is recorded as the reference distribution curve. Abnormal protruding parts corresponding to the reference distribution curve are recorded as reference protruding parts. The symmetrical outlier is calculated based on the overlap between the reference protruding part and the abnormal protruding part. The smaller the overlap, the larger the symmetrical outlier. S43: When the independently identified part corresponds to two target distribution curves, calculate the relative distance between the two target distribution curves and record it as the width value, and obtain the distribution curve of the width value along the center line of the skeleton and record it as the width distribution curve; S44: Construct a fitted straight line based on multiple sampling points on the width distribution curve, denoted as the width straight line. The part of the width distribution curve located above the width straight line is denoted as the width abnormal part. Remove the width abnormal parts whose length is less than the preset length threshold. Calculate the width abnormal value based on the average offset of the width value of the width abnormal part relative to the width straight line. Set the width anomaly value of the independent identification portion of a target distribution curve with only one target distribution curve to 0.

6. The method for detecting abnormal product retrieval behavior of an unmanned vending machine according to claim 1, characterized in that, The process of obtaining the skeleton coordinate combination is as follows: Bind the occluded sensitive area to its corresponding skeleton centerline, and assign a positioning coordinate to each skeleton centerline with the head vertex in the skeleton coordinate system as the reference point. The corresponding one-dimensional coordinate axis is determined based on the center line of the skeleton, with the positive direction of the coordinate axis pointing away from the reference point. The node coordinates are determined based on the distribution range of the occlusion sensitive area on the corresponding center line of the skeleton. The skeleton coordinate combination is obtained by combining the positioning coordinates and the node coordinates.

7. The method for detecting abnormal product retrieval behavior of an unmanned vending machine according to claim 6, characterized in that, Distortion correction is performed on the node coordinates to compensate for the differences in unit quantities of the skeleton coordinate system caused by different shooting distances. The correction process is as follows: P1: A ground reference coordinate system is preset. The key point of the toe in the skeleton coordinate system is recorded as the positioning anchor point. Based on the coordinates of the positioning anchor point in the ground reference coordinate system, the positioning coordinates are recorded. A three-dimensional spatial coordinate system is constructed. The skeleton center line corresponding to the lower leg is represented in the three-dimensional spatial coordinate system and recorded as the anchoring center line. The anchoring center line is perpendicular to the ground reference coordinate system. A plane perpendicular to the horizontal component of the camera shooting direction is drawn through the anchoring center line and recorded as the anchoring plane. P2: A preset image pixel space projection model is used to determine the imaging ray corresponding to each pixel in each valid image; Based on the endpoint positions of each skeleton centerline in the effective image, the endpoint rays of each skeleton centerline are determined. The intersection of the endpoint rays and the anchoring plane is recorded as the spatial projection point of the skeleton centerline endpoint, thus obtaining the actual expression of the skeleton coordinate system in the three-dimensional spatial coordinate system, and representing the coordinates of each node in the three-dimensional coordinate system. P3: Select a skeleton centerline as the basic reference length to establish skeleton proportional constraints. Based on the symmetry of the skeleton centerlines in the skeleton coordinate system, correct the lengths of abnormally long skeleton centerlines while maintaining their direction. P4: The skeleton centerline that needs to be length corrected in P3 is called the correction centerline. Based on the proportional coordinates of the node coordinates on the correction centerline, the corresponding coordinate positions are marked on the skeleton centerline after length correction. Using the skeleton coordinate system in the anchoring plane as the reference coordinate system, obtain the coordinates of each node and regenerate the skeleton coordinate combination.

8. The method for detecting abnormal product retrieval behavior of an unmanned vending machine according to claim 7, characterized in that, The correction conditions are as follows: Rule 1: Make the center lines of the two symmetrical skeletons have the same length, and the length is equal to the maximum of the two lengths; Rule 2: Calculate the lower limit of the length corresponding to each skeleton centerline based on the basic reference length. When the length of any skeleton centerline is less than the lower limit, set its length to be equal to the lower limit.

9. The method for detecting abnormal product retrieval behavior of an unmanned vending machine according to claim 1, characterized in that, The disappearance determination process is as follows: The times when the weight sensor values ​​change corresponding to the removal and return actions are recorded as the removal time and the return time, respectively, and the time intervals corresponding to the removal time and the return time are recorded as the replacement interval. The unit time interval is obtained by dividing the minimum width of the goods of equal weight by the preset speed reference value. Multiple picking images in the replacement interval are extracted using the unit time interval as the extraction interval. The goods of equal weight in the picking images are identified. Picking images with goods of equal weight are recorded as existing images and assigned a value of 1. Picking images without goods of equal weight are recorded as missing images and assigned a value of 0. Construct a time series of pickup images. If there is a partial sequence in the time series of pickup images that satisfies the condition that the number of images is greater than a preset threshold and all elements in the sequence are 0, then extract multiple existing images that are adjacent to the starting element of the partial sequence and record them as target images. Based on the decreasing contour variation pattern of equally heavy objects in the target image, a disappearance trajectory of equally heavy objects is constructed. When the disappearance trajectory determines the disappearance area corresponding to the equally heavy object, if the disappearance area belongs to the occlusion sensitive area, an abnormal replacement signal is generated.

10. A method for detecting abnormal product retrieval behavior in an unmanned vending machine according to claim 9, characterized in that, The process for determining the vanishing region is as follows: The outlines of objects of equal weight in each target image are recorded as visual outlines, and the completeness of each visual outline is determined by combining the complete outlines of the objects of equal weight. Visual contours with a completeness lower than a preset completeness threshold are denoted as partial contours. Based on the prior knowledge of the complete contours of medium-weight objects in the image, contour extraction, missing segment localization, and shape prior completion are performed on the partial contours. The complete mask of the corresponding object is output, and the complete mask area corresponding to the partial contour with the lowest completeness is denoted as the disappearing area.

Images