A multi-modal intelligent cargo cabinet anomaly detection method and system based on visible light and infrared

Abstract

The application relates to the technical field of intelligent cabinets, in particular to a multi-modal intelligent cabinet abnormality detection method based on visible light and infrared, which comprises the following steps: collecting images through a sensor assembly; analyzing image data collected by the sensor assembly and combining a detection algorithm to obtain images of a hand entering an intelligent cabinet and a hand leaving the intelligent cabinet; judging whether the hand holds an article in each stage and analyzing the type of the article through the images of the hand entering the intelligent cabinet and the hand leaving the intelligent cabinet to obtain a behavior classification result; when a target classification result of the object fails, calling data of the sensor assembly to review a disputed process; and comparing an RFID reading result of the intelligent cabinet with the behavior classification result. Through a multi-modal intelligent detection algorithm corresponding to visible light and infrared cameras, abnormal conditions occurring when the intelligent cabinet is used can be automatically detected, and the robustness of the intelligent cabinet can be supported.

Description

Technical Field

[0001] This invention relates to the field of smart vending machine technology, specifically to a multimodal smart vending machine anomaly detection method and system based on visible light and infrared. Background Technology

[0002] Smart vending machines are intelligent warehousing terminal devices that integrate radio frequency identification (RFID), Internet of Things (IoT) sensing technology and embedded control systems. In actual use, due to missing RFID tags, storage of unregistered items, and RFID tags being obscured during stacking, the actual items stored in the vending machine often do not match the information recorded by the system.

[0003] To address the aforementioned issues, relevant patents have proposed corresponding technical solutions. Patent CN202511291387.5 discloses an intelligent control method for automatic guided conveying and stacking, which can solve the placement problem of extra-large containers and avoid label obstruction caused by equipment stacking. However, this method is not applicable to small and medium-sized intelligent containers and fails to solve the problems of missing RFID tags and the storage of unregistered items. Patents CN202511376760.7 and CN202511503877.7 use cameras to acquire image information to detect the situation inside the container, but they still cannot handle the stacking of items and the detection of unregistered items. Patent CN111428822A verifies the storage of items by installing cameras on each layer, but this method requires the deployment of multiple camera sensors and cannot identify items of unidentified types.

[0004] In addition, existing technologies also employ manual inspection methods, where staff periodically check the status of items in the vending machine and the validity of the RFID tags attached to each item. This method significantly increases the maintenance cost of the equipment and also reduces the intelligence and reliability of the smart vending machine. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a multimodal intelligent vending machine anomaly detection method and system based on visible light and infrared. By using a multimodal intelligent detection algorithm corresponding to visible light and infrared cameras, the method automatically detects abnormal situations that occur during the use of the intelligent vending machine, thereby providing support for the robustness of the intelligent vending machine.

[0006] According to one aspect of the present invention, a multimodal intelligent vending machine anomaly detection method based on visible light and infrared is provided, comprising the following steps:

[0007] Install sensor components and acquire images through the sensor components;

[0008] By analyzing the image data collected by the sensor components and combining it with the detection algorithm, images of the hand entering and leaving the smart vending machine are obtained.

[0009] By analyzing images of hands entering and leaving the smart vending machine, we can determine whether the hands are holding items at each stage and analyze the type of items to obtain behavior classification results.

[0010] When the target classification result of an object fails, the data from the sensor components is retrieved to review the disputed process.

[0011] The RFID reading results of the smart vending machine are compared with the behavior classification results. If the type of item to be picked up or put down is detected, the operation type is verified to be consistent with the RFID record. If the system does not detect the type of item to be picked up or put down, the RFID record is verified to show whether there is an operation to pick up or put down an item.

[0012] Furthermore, the sensor assembly includes a visible light camera and an infrared camera, both with parallel detection directions and installed either horizontally or vertically.

[0013] Furthermore, by analyzing the image data collected by the sensor components and combining it with the detection algorithm, images of the hand entering and leaving the smart vending machine are obtained, including:

[0014] Collect visible light video data and sample the video data at fixed time intervals to generate an image sequence;

[0015] By combining the ORB detection algorithm, feature points in the image are extracted in real time, and the coordinates and feature vectors of each feature point are saved.

[0016] The mapping relationship between feature points in two adjacent frames is calculated to achieve feature point matching;

[0017] For completed feature point matching pairs, the changes in feature point coordinates are analyzed to estimate the hand movement state and direction.

[0018] Furthermore, for the completed feature point matching pairs, an analysis of the changes in feature point coordinates is performed, including:

[0019] Calculate the change of each feature point with respect to the X coordinate;

[0020] The results for each pair of feature points are statistically analyzed sequentially.

[0021] By extracting and statistically analyzing the Y-axis features of all filtered feature points, three state counters are obtained: the upward movement counter CountT, the leftward movement counter CountB, and the stationary state counter CountC. The state with the largest counter value is the current state.

[0022] Furthermore, by analyzing images of hands entering and leaving the smart vending machine, we determine whether the hands are holding items at each stage and analyze the type of items to obtain behavioral classification results, including:

[0023] Using the Laplace variance method, images of the hand entering and leaving the smart vending machine are calculated separately, sorted according to the variance, and the top N images with the highest scores for the hand entering and leaving the smart vending machine are retained respectively.

[0024] Two images are extracted from the hand entry stage and the hand exit stage respectively. The moving area in the image is extracted using the inter-frame difference method. The area is then analyzed. If only the hand is present, the stage is the empty-hand mode; otherwise, it is the mode of holding an item.

[0025] When determining the held item in the second step, N images are individually fed into the object recognition model. By analyzing the visible parts of the item and using an image classification strategy based on multi-instance learning, the category of the held item is determined.

[0026] Furthermore, when the object classification result fails, data from the sensor components is retrieved to review the disputed process, including:

[0027] Obtain the first visible light image;

[0028] Obtain the second visible light image;

[0029] The foreground region is obtained by difference;

[0030] Extract the infrared image of the foreground;

[0031] Clustering of infrared image thresholds;

[0032] Determine if the distance between two cluster centers is less than a threshold;

[0033] If the judgment is yes, it means that there is no item in hand; if not, it means that there is an item in hand.

[0034] According to one aspect of the present invention, a multimodal intelligent container anomaly detection system based on visible light and infrared is provided, comprising:

[0035] The first main module is used to acquire images through sensor components;

[0036] The second main module analyzes the image data collected by the sensor components and combines it with the detection algorithm to obtain images of the hand entering the smart vending machine and the hand leaving the smart vending machine.

[0037] The third main module analyzes images of hands entering and leaving the smart vending machine, determines whether the hands are holding items at each stage and analyzes the type of items to obtain behavior classification results.

[0038] The fourth main module calls on the sensor components to review the disputed process when the object classification result fails.

[0039] The fifth main module compares the RFID reading results of the smart vending machine with the behavior classification results. If the type of item being picked up or put down is detected, it verifies whether the operation type is consistent with the RFID record. If the system does not detect the type of item being picked up or put down, it verifies whether there is an operation of picking up or putting down items in the RFID record.

[0040] According to one aspect of the present invention, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of a multimodal intelligent container anomaly detection method based on visible light and infrared.

[0041] According to one aspect of the present invention, a non-transitory computer read storage medium is provided, the non-transitory computer read storage medium storing computer instructions that cause the computer to perform the steps of a multimodal intelligent container anomaly detection method based on visible light and infrared.

[0042] Compared with the prior art, the technical solution of this application has the following beneficial effects:

[0043] This multimodal intelligent container anomaly detection method and system based on visible light and infrared utilizes multimodal data from only one visible light camera and one infrared camera, along with the addition of an intelligent analysis system, to improve the robustness of intelligent containers. It avoids detection omissions caused by RFID tag detachment, storage of unregistered items, and abnormal stacking, reduces the number of manual inspection steps for container items, significantly increases equipment maintenance costs, and diminishes the intelligence and reliability of intelligent containers. Attached Figure Description

[0044] Figure 1 This is a schematic diagram showing the positions of the two sensors in this invention;

[0045] Figure 2 This is a schematic diagram of the sensor installation layout of the present invention;

[0046] Figure 3 This is a schematic diagram of feature point matching according to the present invention;

[0047] Figure 4 This is a flowchart of the feature point Y-axis coordinate statistical method of the present invention;

[0048] Figure 5 This is a schematic diagram of the Laplace variance of the present invention;

[0049] Figure 6 This is a schematic diagram of the review process of the present invention;

[0050] Figure 7 This is a schematic diagram showing the installation location of the RFID reader of the present invention;

[0051] Figure 8 This is a schematic diagram of the detection process of the present invention;

[0052] Figure 9 This is a schematic diagram of the detection system of the present invention. Detailed Implementation

[0053] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] Please see Figure 1-9 This embodiment presents a multimodal intelligent vending machine anomaly detection method and system based on visible light and infrared, comprising the following steps:

[0055] S1. Install the sensor assembly and acquire images through the sensor assembly;

[0056] S2. Analyze the image data collected by the sensor components and combine it with the detection algorithm to obtain images of the hand entering the smart vending machine and the hand leaving the smart vending machine.

[0057] S3. Analyze the images of the hand entering and leaving the smart vending machine, determine whether the hand is holding an item at each stage and analyze the type of item to obtain the behavior classification results;

[0058] S4. When the target classification result of the object fails, call the data of the sensor component to review the disputed process.

[0059] S5. Compare the RFID reading results of the smart vending machine with the behavior classification results. If the type of item to be picked up or put down is detected, verify whether the operation type is consistent with the RFID record. If the system does not detect the type of item to be picked up or put down, verify whether the RFID record shows an operation to pick up or put down an item.

[0060] The sensor assembly includes a visible light camera and an infrared camera, whose detection directions are parallel to each other, and are installed as follows: Figure 1The diagram shows either a horizontal or vertical installation, with the sensor assembly mounted on the top of the smart vending machine, its optical axis pointing vertically downwards. Their positional relationship is as follows: Figure 2 As shown.

[0061] This method is used to detect abnormal situations during the storage and retrieval process of smart vending machines. It aims to analyze the type and status of stored and retrieved items using visible light and infrared cameras, combined with detection algorithms, and compare the results with the detection results of the smart vending machine's RFID. When a mismatch occurs, a reminder message can be sent to the operator to verify the current operation.

[0062] In this embodiment of the invention, image data collected by sensor components is analyzed and combined with a detection algorithm to obtain images of the hand entering and leaving the smart vending machine, including:

[0063] Visible light video data is collected, and the video data is sampled at fixed time intervals (∆T) to generate an image sequence;

[0064] By combining feature point detection algorithms such as ORB in computer vision, feature points in images are extracted in real time, and the coordinates and feature vectors of each feature point are saved.

[0065] Calculate the mapping relationship between feature points in two adjacent frames to complete feature point matching;

[0066] For the completed feature point matching pairs, the changes in feature point coordinates are analyzed to estimate the hand movement state and direction.

[0067] The analysis includes the change in feature point coordinates, including:

[0068] The change of each feature point with respect to the X coordinate is calculated, and its functional relationship expression is as follows;

[0069] ;

[0070] The results for each pair of feature points are statistically analyzed sequentially.

[0071] like Figure 4 As shown, by extracting and statistically analyzing the Y-axis features of all filtered feature points, three state counters are obtained: the upward movement counter CountT, the leftward movement counter CountB, and the stationary state counter CountC. The state with the largest counter value is the current state.

[0072] In this embodiment of the invention, behavior classification aims to analyze images of hands entering and leaving the smart vending machine. By analyzing whether the hand is holding an item at each stage and the type of item, the video information is used to determine whether the current operation involves storing or retrieving an item, and the type of item involved. This includes the following steps:

[0073] 1. Filtering for clear images;

[0074] Using the Laplace variance method, images of the hand entering and leaving the smart vending machine are calculated separately, sorted according to the variance, and the top N images with the highest scores for the hand entering and leaving the smart vending machine are retained respectively.

[0075] The specific method is as follows:

[0076] 1) Define the Laplacian operator and compute the convolution;

[0077] The Laplace operator is:

[0078] ;

[0079] The method for calculating convolution is as follows:

[0080] ;

[0081] Where I is the input image data, K is the Laplacian operator, i and j are the index numbers of the parameters in the Laplacian operator, and x and y are the pixel coordinates of the image to be calculated.

[0082] 2) Calculate the mean of the response plot;

[0083] The arithmetic mean μ of all pixel values ​​in the Laplacian response map L is calculated as follows:

[0084] ;

[0085] Where L(x, y) is the feature map after Laplacian transformation, x and y are the pixel coordinates of the feature map, and M and N are the vertical and horizontal resolutions of the feature map, respectively.

[0086] 3) Calculate the Laplace variance;

[0087] Calculate the variance σ of the response plot L. 2 This value is the image sharpness score, calculated as follows:

[0088] ;

[0089] Where L(x, y) is the feature map after Laplace transform, x and y are the pixel coordinates of the feature map, M and N are the vertical and horizontal resolutions of the feature map, and μ is the pixel mean of the feature map.

[0090] If the image is clear and the grayscale changes drastically at the edges, the calculated score will be larger; conversely, if the image is blurry and the edges are smooth, the calculated score will be smaller.

[0091] By calculating the Laplacian difference of an image sequence, blurry images can be automatically removed, thus avoiding interference from blurry images with subsequent processing results.

[0092] 2. Determine whether you are holding any items in your hand;

[0093] Two images are extracted from the hand entry stage and the hand exit stage respectively. The moving area in the image is extracted using the inter-frame difference method. The area is then analyzed. If only the hand is present, the stage is the empty-hand mode; otherwise, it is the mode of holding an item.

[0094] 1) Calculation of absolute difference;

[0095] Calculate the absolute grayscale difference between two frames of images at pixel position (x, y);

[0096] ;

[0097] Among them, I t Let x and y be the image data at time t, where x and y are the pixel coordinates of the image, respectively.

[0098] 2) Binarization threshold segmentation;

[0099] The difference map is converted into a binary mask to distinguish between moving regions and static backgrounds;

[0100] ;

[0101] Where D(x, y) is the absolute grayscale difference map of the image, x and y are the pixel coordinates of the difference map respectively, and T is a preset threshold.

[0102] 3) Whether or not you possess any items;

[0103] The color of the hand and the shape of the grip are quite obvious. When a noticeable bulge is detected in the moving part in the horizontal or vertical direction, or when there is a significant difference in color between the moving part in front or in the vertical direction, it indicates that the hand is holding an item.

[0104] C shape D represents the rate of change of the maximum convex hull defect depth or projected width of the hand contour in the horizontal or vertical direction. colorThe method for determining the color histogram differences (such as Bartholin's distance) or variance in the front or depth directions of the hand area is as follows:

[0105] ;

[0106] Here, H=1 indicates that a handheld item has been detected, w1 and w2 are the weighting coefficients for shape and color, respectively, and τ is the decision threshold used to filter noise.

[0107] 3. Item Classification:

[0108] When determining the held item in the second step, N images are individually fed into the object recognition model. By analyzing the visible parts of the item and using an image classification strategy based on multi-instance learning, the category of the held item is determined.

[0109] 1) Single-frame detection:

[0110] For each image, run the detector and extract the probability of the category with the highest confidence level in the image as the "voting power" for that image;

[0111] ;

[0112] Among them, s n,k The score of the nth image supporting the kth class is I. n This is the input image data for the nth image;

[0113] At the same time, the score of each classifier is judged, and when the classification score is lower than the set threshold, the current classification result is determined to be invalid.

[0114] 2) Voting aggregation:

[0115] The scores of all N images are averaged (soft voting) to obtain the final total score for each category;

[0116] ;

[0117] Among them, V k R is the average confidence score of class k across all images. n,k Let n be the confidence score of the nth image belonging to the kth class.

[0118] 3) Final decision:

[0119] The category with the highest score that exceeds the threshold is selected as the final result;

[0120] ;

[0121] Among them, V k τ is the average confidence score of the k-th class across all images, and τ is a pre-set confidence threshold.

[0122] When the total result value is below the threshold, the object classification is considered to have failed, and the system has failed to identify the type of object due to reasons such as angle.

[0123] ;

[0124] Where Result is the type result calculated by the system, and Threshold is... Score ClassifyFailt is a marker indicating a failed classification, set by a pre-defined confidence score threshold for the system.

[0125] This method also includes:

[0126] 4. Empty-handed assessment and verification:

[0127] When the object classification result fails, data from the infrared camera is retrieved to review the disputed process. The review method is as follows:

[0128] Obtain the first visible light image;

[0129] Obtain the second visible light image;

[0130] The foreground region is obtained by difference;

[0131] Extract the infrared image of the foreground;

[0132] Clustering of infrared image thresholds;

[0133] Determine if the distance between two cluster centers is less than a threshold;

[0134] If the judgment is yes, it means that there is no item in hand; if not, it means that there is an item in hand.

[0135] Mission complete.

[0136] 5. Anomaly Detection:

[0137] Anomaly detection refers to acquiring the results of RFID detection and comparing them with the behavior classification results from the third step, including:

[0138] If the system detects the type of item being retrieved or placed, verify that the operation type matches the RFID record;

[0139] If the system does not detect the type of item being picked up or placed, it will verify whether there is an item picking or placing operation in the RFID record;

[0140] When the two results contradict each other, the operator should be notified to conduct a timely verification.

[0141] It should be noted that RFID card readers are generally installed on both sides of the smart cabinet, with specific locations as follows: Figure 7 As shown.

[0142] The specific operation is as follows: After each door is closed, the smart tool cabinet activates the reader to read the internal tool information. When a new door closing action occurs, it reads the current tool status and compares it with the previous information to deduce the type of item and the storage / retrieval operation. At the same time, it uses this method to obtain the current operation type and the target item. If the two contradict each other, the operator is notified on the spot for manual verification to avoid misoperation.

[0143] The implementation of the various embodiments of the present invention is based on programmed processing by a device with processor functionality. Therefore, in practical engineering, the technical solutions and functions of the various embodiments of the present invention are encapsulated into various modules. Based on this reality, and building upon the above embodiments, the embodiments of the present invention provide a multimodal intelligent vending machine anomaly detection system based on visible light and infrared, which is used to execute a multimodal intelligent vending machine anomaly detection method based on visible light and infrared from the above method embodiments.

[0144] See Figure 9 The system includes: a first main module for acquiring images via sensor components; a second main module for analyzing the image data acquired by the sensor components and combining it with detection algorithms to obtain images of the hand entering and leaving the smart vending machine; a third main module for analyzing the images of the hand entering and leaving the smart vending machine, determining whether the hand is holding an item at each stage and analyzing the type of item to obtain a behavior classification result; a fourth main module for re-examining the disputed process by calling the data from the sensor components when the target classification result of the object fails; and a fifth main module for comparing the RFID reading result of the smart vending machine with the behavior classification result. If the type of item being picked up or placed is detected, the module verifies whether the operation type is consistent with the RFID record. If the system does not detect the type of item being picked up or placed, the module verifies whether the RFID record shows an operation of picking up or placing an item.

[0145] The multimodal intelligent container anomaly detection system based on visible light and infrared provided in this embodiment of the invention adopts... Figure 9 Several modules within the system utilize multimodal intelligent detection algorithms corresponding to visible light and infrared cameras to automatically detect abnormal situations that occur during the use of the smart vending machine, thus providing support for the robustness of the smart vending machine.

[0146] It should be noted that the system embodiments provided by this invention, in addition to implementing the methods in the above method embodiments, are also used to implement the methods in other method embodiments provided by this invention. The difference lies only in setting corresponding functional modules, and their principles are basically the same as those of the above system embodiments provided by this invention. As long as those skilled in the art, based on the above system embodiments and referring to the specific technical solutions in other method embodiments, obtain corresponding technical means and technical solutions composed of these technical means by combining technical features, and improve the modules in the above system embodiments while ensuring the practicality of the technical solutions, they can obtain corresponding system-like embodiments for implementing the methods in other method-like embodiments. For example:

[0147] Based on the above system embodiments, as a preferred embodiment, the multimodal intelligent container anomaly detection system based on visible light and infrared provided in this embodiment of the invention includes a visible light camera and an infrared camera, the detection directions of which are parallel to each other, and the installation method is either left-right or top-bottom installation.

[0148] Based on the above system embodiments, as a preferred embodiment, the multimodal smart vending machine anomaly detection system based on visible light and infrared provided in this embodiment analyzes the image data collected by the sensor components and combines it with a detection algorithm to obtain images of the hand entering and leaving the smart vending machine, including:

[0149] Collect visible light video data and sample the video data at fixed time intervals to generate an image sequence;

[0150] By combining the ORB detection algorithm, feature points in the image are extracted in real time, and the coordinates and feature vectors of each feature point are saved.

[0151] The mapping relationship between feature points in two adjacent frames is calculated to achieve feature point matching;

[0152] For completed feature point matching pairs, the changes in feature point coordinates are analyzed to estimate the hand movement state and direction.

[0153] Based on the above system embodiments, as a preferred embodiment, the multimodal intelligent container anomaly detection system based on visible light and infrared provided in this embodiment of the invention analyzes the changes in feature point coordinates of completed feature point matching pairs, thereby estimating the hand movement state and direction, including:

[0154] Calculate the change of each feature point with respect to the X coordinate;

[0155] The results for each pair of feature points are statistically analyzed sequentially.

[0156] By extracting and statistically analyzing the Y-axis features of all filtered feature points, three state counters are obtained, and the state with the largest counter value is the current state.

[0157] Based on the above system embodiments, as a preferred embodiment, the multimodal smart vending machine anomaly detection system based on visible light and infrared provided in this embodiment analyzes images of hands entering and leaving the smart vending machine, determines whether the hand is holding an item at each stage, analyzes the type of item, and obtains behavior classification results, including:

[0158] Using the Laplace variance method, images of the hand entering and leaving the smart vending machine are calculated separately, sorted according to the variance, and the top N images with the highest scores for the hand entering and leaving the smart vending machine are retained respectively.

[0159] Two images are extracted from the hand entry stage and the hand exit stage respectively. The moving area in the image is extracted using the inter-frame difference method. The area is then analyzed. If only the hand is present, the stage is the empty-hand mode; otherwise, it is the mode of holding an item.

[0160] When determining the held item in the second step, N images are individually fed into the object recognition model. By analyzing the visible parts of the item and using an image classification strategy based on multi-instance learning, the category of the held item is determined.

[0161] Based on the above system embodiments, as a preferred embodiment, the multimodal intelligent vending machine anomaly detection system based on visible light and infrared provided in this embodiment of the invention, when the target classification result of an object fails, calls the data of the sensor components to review the disputed process, including:

[0162] Obtain the first visible light image;

[0163] Obtain the second visible light image;

[0164] The foreground region is obtained by difference;

[0165] Extract the infrared image of the foreground;

[0166] Clustering of infrared image thresholds;

[0167] Determine if the distance between two cluster centers is less than a threshold;

[0168] If the judgment is yes, it means that there is no item in hand; if not, it means that there is an item in hand.

[0169] Based on the same inventive concept as the foregoing embodiments, this embodiment of the invention also provides a computer device, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of a multimodal intelligent container anomaly detection method based on visible light and infrared light, as shown below:

[0170] Images are acquired using sensor components;

[0171] By analyzing the image data collected by the sensor components and combining it with the detection algorithm, images of the hand entering and leaving the smart vending machine are obtained.

[0172] By analyzing images of hands entering and leaving the smart vending machine, we can determine whether the hands are holding items at each stage and analyze the type of items to obtain behavior classification results.

[0173] When the target classification result of an object fails, the data from the sensor components is retrieved to review the disputed process.

[0174] The RFID reading results of the smart vending machine are compared with the behavior classification results. If the type of item to be picked up or put down is detected, the operation type is verified to be consistent with the RFID record. If the system does not detect the type of item to be picked up or put down, the RFID record is verified to show whether there is an operation to pick up or put down an item.

[0175] Based on the same inventive concept as the foregoing embodiments, this embodiment of the invention also provides a non-transitory computer read storage medium that stores computer instructions. These computer instructions cause the computer to execute the steps of a multimodal intelligent vending machine anomaly detection method based on visible light and infrared light, as shown below:

[0176] Images are acquired using sensor components;

[0177] By analyzing the image data collected by the sensor components and combining it with the detection algorithm, images of the hand entering and leaving the smart vending machine are obtained.

[0178] By analyzing images of hands entering and leaving the smart vending machine, we can determine whether the hands are holding items at each stage and analyze the type of items to obtain behavior classification results.

[0179] When the target classification result of an object fails, the data from the sensor components is retrieved to review the disputed process.

[0180] The RFID reading results of the smart vending machine are compared with the behavior classification results. If the type of item to be picked up or put down is detected, the operation type is verified to be consistent with the RFID record. If the system does not detect the type of item to be picked up or put down, the RFID record is verified to show whether there is an operation to pick up or put down an item.

[0181] In summary, the present invention utilizes multimodal data from only one visible light camera and one infrared camera, along with the addition of an intelligent analysis system, to improve the robustness of the smart container, avoid issues such as RFID tag detachment, storage of unregistered items, and detection omissions caused by abnormal stacking, reduce the number of manual inspection steps for container items, significantly increase equipment maintenance costs, and reduce the intelligence and reliability of the smart container.

[0182] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0183] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A multimodal intelligent vending machine anomaly detection method based on visible light and infrared, characterized in that, Includes the following steps: Install sensor components and acquire images through the sensor components; By analyzing the image data collected by the sensor components and combining it with the detection algorithm, images of the hand entering and leaving the smart vending machine are obtained. By analyzing images of hands entering and leaving the smart vending machine, we can determine whether the hands are holding items at each stage and analyze the type of items to obtain behavior classification results. When the target classification result of an object fails, the data from the sensor components is retrieved to review the disputed process. The RFID reading results of the smart vending machine are compared with the behavior classification results. If the type of item to be picked up or put down is detected, the operation type is verified to be consistent with the RFID record. If the system does not detect the type of item to be picked up or put down, the RFID record is verified to show whether there is an operation to pick up or put down an item.

2. The anomaly detection method for a multimodal intelligent vending machine based on visible light and infrared as described in claim 1, characterized in that: The sensor assembly includes a visible light camera and an infrared camera, both with parallel detection directions and can be installed horizontally or vertically.

3. The anomaly detection method for a multimodal intelligent vending machine based on visible light and infrared as described in claim 2, characterized in that: Analyzing the image data collected by the sensor components and combining it with the detection algorithm, images of the hand entering and leaving the smart vending machine are obtained, including: Collect visible light video data and sample the video data at fixed time intervals to generate an image sequence; By combining the ORB detection algorithm, feature points in the image are extracted in real time, and the coordinates and feature vectors of each feature point are saved. The mapping relationship between feature points in two adjacent frames is calculated to achieve feature point matching; For completed feature point matching pairs, the changes in feature point coordinates are analyzed to estimate the hand movement state and direction.

4. The anomaly detection method for a multimodal intelligent vending machine based on visible light and infrared as described in claim 3, characterized in that: For completed feature point matching pairs, the changes in feature point coordinates are analyzed to estimate the hand movement state and direction, including: Calculate the change of each feature point with respect to the X coordinate; The results for each pair of feature points are statistically analyzed sequentially. By extracting and statistically analyzing the Y-axis features of all filtered feature points, three state counters are obtained: the upward movement counter CountT, the leftward movement counter CountB, and the stationary state counter CountC. The state with the largest counter value is the current state.

5. The anomaly detection method for a multimodal intelligent vending machine based on visible light and infrared as described in claim 1, characterized in that: Analyze images of hands entering and leaving the smart vending machine to determine whether the hand is holding an item at each stage and analyze the type of item to obtain behavior classification results, including: Using the Laplace variance method, images of the hand entering and leaving the smart vending machine are calculated separately, sorted according to the variance, and the top N images with the highest scores for the hand entering and leaving the smart vending machine are retained respectively. Two images are extracted from the hand entry stage and the hand exit stage respectively. The moving area in the image is extracted using the inter-frame difference method. The area is then analyzed. If only the hand is present, the stage is the empty-hand mode; otherwise, it is the mode of holding an item. When determining the held item in the second step, N images are individually fed into the object recognition model. By analyzing the visible parts of the item and using an image classification strategy based on multi-instance learning, the category of the held item is determined.

6. The anomaly detection method for a multimodal intelligent vending machine based on visible light and infrared as described in claim 1, characterized in that: When the object classification result fails, data from the sensor components is retrieved to review the disputed process, including: Obtain the first visible light image; Obtain the second visible light image; The foreground region is obtained by difference; Extract the infrared image of the foreground; Clustering of infrared image thresholds; Determine if the distance between two cluster centers is less than a threshold; If the judgment is yes, it means that there is no item in hand; if not, it means that there is an item in hand.

7. A multimodal intelligent container anomaly detection system based on visible light and infrared, characterized in that, include: The first main module is used to acquire images through sensor components; The second main module analyzes the image data collected by the sensor components and combines it with the detection algorithm to obtain images of the hand entering the smart vending machine and the hand leaving the smart vending machine. The third main module analyzes images of hands entering and leaving the smart vending machine, determines whether the hands are holding items at each stage and analyzes the type of items to obtain behavior classification results. The fourth main module calls on the sensor components to review the disputed process when the object classification result fails. The fifth main module compares the RFID reading results of the smart vending machine with the behavior classification results. If the type of item being picked up or put down is detected, it verifies whether the operation type is consistent with the RFID record. If the system does not detect the type of item being picked up or put down, it verifies whether there is an operation of picking up or putting down items in the RFID record.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the multimodal intelligent container anomaly detection method based on visible light and infrared as described in any one of claims 1 to 6.

9. A non-transitory computer read storage medium, characterized in that, The non-transitory computer read storage medium stores computer instructions, which cause the computer to execute the steps of the multimodal intelligent container anomaly detection method based on visible light and infrared as described in any one of claims 1 to 6.

Images