A method for intelligent recognition of commodity images in vending machines

Abstract

The present invention relates to the technical field of vending machines, and specifically to a method for intelligently identifying commodity images in vending machines, comprising the following steps: Step 1: installing a camera inside the vending machine and performing spatial calibration on the camera; Step 2: the camera scans the commodities on the shelf and generates an initial commodity status table; Step 3: the camera detects the commodity status in real time and obtains commodity images, extracts the status information of all commodities in each frame image, compares the commodity status information in the current frame image with the commodity status information in the previous frame image, determines whether any commodity has been taken out, put back or adjusted, and outputs a real-time commodity status table. This solution can track user interaction records through feature data generated by a classification model, dynamically update the real-time commodity status table in the vending machine system, and provide accurate protection for commodity inventory management; and through an error correction model, the camera can more accurately identify commodity images and reduce recognition statistical errors.

Description

Technical Field

[0001] The present invention relates to the technical field of vending machines, and in particular to a method for intelligently identifying commodity images of vending machines. Background Art

[0002] With the rapid development of artificial intelligence and computer vision technology, vending machine systems based on intelligent recognition of product images have become an important innovation direction in the field of unmanned retail. Through image recognition technology, vending machines can automatically identify and manage inventory of products, thereby improving the user shopping experience and simplifying the management process. However, in practical applications, systems based on intelligent recognition of product images still face many technical challenges, especially in terms of product sales statistics and dynamic inventory updates. There are problems such as inaccurate recognition and calculation delays, which limit the practicality and promotion of the system.

[0003] After searching, Chinese patent number CN201910670731.X discloses an automatic inventory management system for self-service vending machines based on image recognition, including a self-service vending machine, wherein a plurality of parallel cargo positions are arranged in the self-service vending machine, the self-service vending machine and the camera device are respectively connected to a cloud control system, and the cloud control system is provided with a sample database containing the product name, product number, product price and product atlas of pre-sale products; after each replenishment of the self-service vending machine, the cloud control system controls the camera device to capture the image of the product on the outermost cargo position of the aisle; then the image is associated with the aisle number of the shelf, and the cloud control system performs image recognition to obtain the product name, product number and product price of the corresponding product, and after the product information is associated with the aisle number of the shelf, it is stored in the cloud control system and synchronized to the cloud, so that the local and cloud data are synchronized and consistent, and the inventory of the corresponding aisle of the corresponding shelf of the self-service vending machine is updated;

[0004] The above solution uses the camera to automatically identify the goods through the image recognition module. The sample database takes inventory of the existing inventory data according to the consumption of the goods, and transmits the shipping information through the cloud control system. At the same time, it reduces the burden of inventory management and is safe, reliable, convenient and fast.

[0005] However, during the image recognition process, due to the unpredictable nature of user behavior, it can be difficult to identify products in dynamic scenarios. For example, a user's quick movements or multiple pick-up and placement behaviors when taking or placing products may cause the system to count repeatedly or miss items. In addition, failure to completely remove products or change the placement of products by users can also cause statistical deviations.

[0006] Therefore, a method for intelligent recognition of commodity images in vending machines is proposed to solve the above-mentioned problems. Summary of the invention

[0007] Technical issues solved

[0008] In view of the above-mentioned shortcomings of the prior art, the present invention provides a method for intelligent recognition of commodity images in vending machines, which can effectively solve the problem that the commodity image recognition in the prior art cannot accurately capture the user's behavior and actions, resulting in statistical deviation.

[0009] Technical solution: To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0010] The present invention provides a method for intelligently identifying commodity images in a vending machine, comprising the following steps:

[0011] Step 1: Install a camera inside the vending machine and perform spatial calibration on the camera;

[0012] Step 2: The camera scans the goods on the shelf and generates an initial state table of the goods;

[0013] Step 3: The camera detects the status of the goods in real time and obtains the product image, extracts the status information of all the goods in each frame image, compares the status information of the goods in the current frame image with the status information of the goods in the previous frame image, determines whether any goods are taken out, put back or adjusted, and outputs the real-time status table of the goods;

[0014] Step 4: The camera captures the user's interactive behavior and inputs it into the classification model to obtain feature data , used to dynamically update the real-time status table of commodities;

[0015] Step 5: Feature data Input into the error correction model to obtain the corrected characteristic data , used to eliminate product image recognition errors;

[0016] Step 6: Correct the feature data Input into the user behavior prediction model to obtain the time dependency and behavior pattern of user behavior.

[0017] Furthermore, the method of spatially calibrating the camera in step 1 includes:

[0018] Calibrate the camera to obtain the camera's intrinsic parameter matrix and the external parameter matrix ;

[0019] Define the spatial scope and size parameters of the shelf, including: the total width of the shelf , Total Height , Shelf depth ; Divide the shelf into grid units of uniform size and obtain the center point coordinates of the grid units ;

[0020] Based on the size of the shelf, the internal parameter matrix and the external parameter matrix The pixel coordinates on the image acquired by the camera Map it to the world coordinate system to get the three-dimensional position of the midpoint in the world coordinate system ;

[0021] The 3D position of the point in the world coordinate system Input into the optimization model to obtain the optimization results of the camera internal and external parameter calibration and the optimization results of the midpoint of the world coordinate system.

[0022] Furthermore, the optimization method of the optimization model is:

[0023] Define the system state vector and the model observations are ; Define the prediction model of camera position and calibration environment in a short static state: ; In the formula, represents the predicted state estimate; represents the updated state estimate;

[0024] Using Kalman Gain Fuse the current measured values ​​and predicted values, update the prediction model, ; In the formula, Represents the current observation value; Represents the projection of the predicted value in the observation space; is the observation matrix; is the observation residual; Kalman gain Is a weight matrix, calculated as: ;in, is the prediction error covariance matrix; is the transpose of the observation matrix; Represents the combined uncertainty between predictions and measurements; is the correction amount; is the measurement noise covariance matrix;

[0025] Forecast error covariance matrix Update, the update formula is: ;in, is the updated error covariance matrix; is the identity matrix;

[0026] Input each frame of data into the optimization model and repeat the above steps.

[0027] Furthermore, the method of obtaining the product data in step 2 includes:

[0028] Identify the category of goods through the object detection module and the location of the rectangular bounding box of the product , is the horizontal coordinate value of the leftmost side of the rectangular border. is the vertical coordinate value of the top side of the rectangular border, Represents the total width of the rectangular bounding box, Represents the total height of the rectangular bounding box; extracts the feature description vector of each product through the image processing module ; Product initial status table ;in, is the product depth value, Initialization completion time.

[0029] Furthermore, the method of determining whether any product has been taken out, put back or adjusted in step 3 includes:

[0030] If a product is not detected in the current frame image and its position is within a continuous time period If it is empty, it is determined that the product has been taken out; is the time threshold, determined by the frame rate and the number of consecutive undetected frames Calculated; if a product appears in the current frame image, but does not exist in the previous frame image, it is determined that the product is put back; if the position of the product is slightly offset, that is , and , it is judged as a product position adjustment; , , ;in, is the horizontal position change, is the vertical position change, is the depth position change; is the upper left corner position of the rectangular bounding box of the product in the previous frame image, is the upper left corner position of the rectangular bounding box of the product in the current frame image, is the depth center coordinate of the rectangular bounding box of the product in the previous frame image, is the depth center coordinate of the rectangular bounding box of the product in the current frame image; the judgment threshold includes the horizontal position change threshold , vertical position change threshold and depth position change threshold .

[0031] Furthermore, the method of capturing the user's interactive behavior in step 4 includes:

[0032] The camera captures the user's hand key points based on the human posture estimation model and obtains the user's hand trajectory set , ,in, is the horizontal coordinate of the hand in the image, is the vertical coordinate of the hand in the image, is the depth information of the hand in three-dimensional space, It is the timestamp of the current frame in the entire motion trajectory;

[0033] Based on the user's hand trajectory collection Determine whether the user's hand motion trajectory overlaps with the rectangular boundary box of the product to trigger the product's interactive behavior. The judgment conditions are: and ,in, is the left edge of the rectangular bounding box of the product in the horizontal direction. is the right edge of the rectangular bounding box of the product in the horizontal direction. is the top vertical boundary of the rectangular bounding box of the product. The bottom boundary of the rectangular bounding box of the product in the vertical direction;

[0034] If the camera detects that the hand interacts with the product, the interaction time is recorded , Product category and where the interaction occurs Based on the dynamic relationship between the hand motion trajectory and the rectangular bounding box of the product, the interaction behavior is divided into two types of interaction events:

[0035] Enter the behavior, the trigger condition is in the current frame: or , in the next frame: and ;

[0036] Remove behavior, the trigger condition is in the current frame: and , in the next frame or ;

[0037] Based on interaction events and time series features, the user's hand trajectory is collected The rectangular bounding box of the product changes dynamically Input into the pre-built classification model to identify the user's specific interaction behavior , and generate feature data based on each specific interaction behavior of the user ;in, is the timestamp, Indicates The feature data of the user's interaction with the product is detected. is the type of interactive behavior; interactive behavior Includes: taking out goods, putting goods back, and adjusting goods.

[0038] Furthermore, the triggering condition for taking out the product is:

[0039] In time : ;

[0040] In time : ;

[0041] And the rectangular bounding box of the product is at time disappear: , Indicates at time In the image frame, the object detection module does not detect the rectangular bounding box of the product;

[0042] The triggering conditions for returning the goods are:

[0043] In time : ;

[0044] In time : ;

[0045] And the rectangular bounding box of the product is at time Added: ;

[0046] The triggering conditions for adjusting the product are:

[0047] In time , : and ;

[0048] And the change in product position includes , and Less than the judgment threshold , and .

[0049] Furthermore, the error correction model in step 5 is constructed as follows:

[0050] Define the product recognition errors, including: fast motion capture error, multiple pick-and-place recognition error, product incomplete removal recognition error, product placement change recognition error; define parameter time window ;

[0051] Construct the expression of the error correction model:

[0052] ;

[0053] In the formula, Is a conditional statement used to determine each feature data Should it be included in the correction feature data? Inside; For feature data Total number of; is the time interval, ; is the offset of the center point of the rectangular bounding box of the product, ; is the offset threshold; is the product status of the current frame image, It is the product status of the previous frame image.

[0054] Furthermore, the method of constructing the user behavior prediction model in step 6 includes:

[0055] Continuously modify feature data for each user behavior Constructed as time series data, the shape is ;in, is the number of time steps, For each interaction The number of features; normalize the numerical features;

[0056] Define the LSTM network structure as a user behavior prediction model and transform the time series data Input into the user behavior prediction model to obtain the category probability distribution at each time step ;

[0057] Collect historical correction feature data , use it as the training input set of the user behavior prediction model, and set the number of iterations; initialize the network parameters of the user behavior prediction model, including: the weight matrix of the input gate , the bias vector of the input gate , the weight matrix of the forget gate , the bias vector of the forget gate , the weight matrix of candidate memory , the bias vector of the candidate memory , the weight matrix of the output gate , the bias vector of the output gate ;

[0058] Define the cross entropy loss function of the user behavior prediction model, which is expressed as: ; In the formula, is the value of the cross entropy loss function; is the sample size; is the true label; The probability predicted by the model; is the predicted category probability distribution The logarithm of For Category Quantity;

[0059] The training input set is input into the user behavior prediction model, followed by the embedding layer, LSTM layer, fully connected layer and Softmax layer; the corresponding cross entropy loss function value is calculated, and the error gradient is back-propagated from the output layer to the input layer; according to the error gradient, the Adam optimization algorithm is used to update the network parameters to reduce the value of the loss function; the time series data is repeatedly trained until the user behavior prediction model reaches the preset number of iterations to obtain the time dependency and behavior pattern of the user behavior.

[0060] Furthermore, the LSTM network structure includes an embedding layer, an LSTM layer, a fully connected layer and a Softmax layer; the embedding layer receives the input time series data , the time series data The features at each time step Mapped to high-dimensional feature space to obtain embedded representation ; In the formula, is the embedding representation, indicating the High-dimensional feature vector for each time step; is the input feature, indicating The original feature vector of time steps; is the embedding weight matrix; is the embedding bias vector;

[0061] The LSTM layer receives the output embedding representation of the embedding layer , get the hidden state ;

[0062] Hide the state Input to the fully connected layer, the hidden state of the fully connected layer for each time step Perform linear transformation to obtain the classification vector ; In the formula, It is used to map the hidden state The weight matrix of is the bias vector of the fully connected layer;

[0063] The classification vector Input to the Softmax layer, the Softmax layer performs The classification vector Calculate each category The probability is calculated as ; In the formula, is the input score, indicating the time step When, category A score of is an exponential function used to input scores Perform nonlinear mapping and input scores Mapping to positive range ; is the normalization factor used to normalize all categories Input score Normalized to probability value; Represents the class probability distribution at each time step.

[0064] Beneficial effects: Compared with the prior art, the technical solution provided by the present invention has the following beneficial effects:

[0065] This solution optimizes the spatial calibration of the camera through the optimization model, obtains the optimization results of the camera's internal and external parameters and the optimization results of the midpoint of the world coordinate system, thereby improving the accuracy of the camera's spatial calibration and providing a guarantee for the accuracy of the camera's product recognition;

[0066] This solution can track user interaction records through feature data generated by the classification model, dynamically update the real-time status table of goods in the vending machine system, and provide accurate guarantee for inventory management of goods;

[0067] This solution uses an error correction model to allow the camera to more accurately distinguish between slight adjustments, repeated operations, and significant pick-and-place behaviors of goods, reducing misjudgments caused by slight movement of goods or rapid repeated operations. At the same time, it can supplement records based on time, location, and status changes to avoid omissions, thereby improving the detection accuracy of product image recognition.

[0068] This solution can classify time series data through the user behavior prediction model, and optimize the product layout and warehouse management of the vending machine based on the analysis of user behavior patterns. For example, it can predict which products will have a high demand in a specific time period and replenish them in advance. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0070] Figure 1 Schematic diagram of the process of intelligent recognition method of commodity images in vending machines according to an embodiment of the present invention. DETAILED DESCRIPTION

[0071] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0072] The present invention will be further described below in conjunction with the embodiments.

[0073] Embodiment 1: See attached Figure 1 , this case proposes a method for intelligent recognition of commodity images in vending machines, comprising the following steps:

[0074] Step 1: Install a high frame rate camera (50FPS or higher) inside the vending machine, ensuring that the camera can cover the entire shelf area inside the vending machine; perform spatial calibration on the camera;

[0075] Step 2: The camera scans the goods on the shelf to obtain the product data, and generates the product initial status table based on the product data;

[0076] Step 3: The camera detects the status of the product in real time and obtains the product image. The target detection module extracts the status information of all products in each frame image, compares the status information of the product in the current frame image with the status information of the product in the previous frame image, determines whether any product has been taken out, put back or adjusted, and outputs the change information (i.e., the real-time status table of the product);

[0077] Step 4: The camera captures the user's interactive behavior and inputs it into the pre-built classification model to obtain feature data , used to dynamically update the real-time status table of commodities;

[0078] Step 5: Then the feature data Input into the pre-built error correction model to obtain the corrected feature data , used to eliminate product image recognition errors;

[0079] Step 6: Correct the feature data Input into the pre-built user behavior prediction model to obtain the time dependency and behavior pattern of user behavior.

[0080] Specifically, in this solution, the purpose of spatially calibrating the camera is to establish a spatial mapping relationship between the camera and the shelf inside the vending machine; by calibrating the camera and dividing the standard grid area of ​​the shelf based on the calibration result; the method of spatially calibrating the camera includes:

[0081] Set the camera's internal and external calibration parameters. The internal calibration parameter is used to calibrate the camera's imaging characteristics, that is, to calculate the camera's internal parameters, including focal length, principal point coordinates, and distortion parameters, through an algorithm. The external calibration parameter is used to establish the relationship between the camera's actual position and posture and the object's coordinate system.

[0082] The setting method of the internal reference calibration includes: using a standard chessboard (such as a square size of 20mm×20mm) as a calibration tool. The regular pattern of the chessboard can help the camera recognize geometric features in the image and calculate the imaging parameters of the camera through these features; placing the chessboard within the field of view of the camera, and taking images of the chessboard from multiple different angles (such as 30° tilt angle, 45° tilt angle, vertical shooting), ensuring that the chessboard pattern covers the entire imaging area of ​​the camera, so as to correct the distortion of the entire field of view; taking at least 20 chessboard images to ensure that sufficient data is collected from different angles and positions to improve the accuracy of the calibration; defining the corner points of the chessboard as the points at the junction of black and white squares, and using these points as reference points used in the calibration process; finding all the corner points on the chessboard and determining the pixel coordinates of these corner points , as the input data for subsequent calibration; the camera's intrinsic parameter matrix is ​​calculated using the pixel coordinates of the corner points (image coordinate system) and the real world coordinates (world coordinate system) ; In the formula, and Respectively represent the focal length of the camera in the horizontal and vertical directions; and is the optical center coordinate of the camera, that is, the position of the center point of the image on the pixel plane; correct the distortion parameters of the camera (radial and tangential distortion) and calculate the distortion coefficient ; In the formula , , is the radial distortion coefficient, Indicates the distortion of the pixels closer to the optical center in the radial direction. Indicates distortion in the medium radial range, Indicates the distortion of pixels in the long-distance radial range; and is the tangential distortion coefficient;

[0083] The setting method of external parameter calibration includes: constructing a world coordinate system, taking the lower left corner of the shelf as the origin (the side close to the camera), and establishing a three-dimensional world coordinate system ; Get the camera's external parameter matrix ; Where R is a 3×3 rotation matrix, which represents the orientation (direction and posture) of the camera; , the first column Indicates the direction of the X-axis in the world coordinate system after rotation in the camera coordinate system. The second column Indicates the direction of the Y axis in the world coordinate system after rotation in the camera coordinate system. The third column Represents the direction of the Z axis in the world coordinate system in the camera coordinate system after rotation; the rotation matrix R rotates the point from the world coordinate system to the camera coordinate system, and the calculation formula is: ,in, Represents the coordinates of the point in the camera coordinate system, Represents the coordinates of the point in the world coordinate system; T is a 3×1 translation vector, which represents the position of the camera in the world coordinate system; , represents the position of the camera optical center in the world coordinate system, is the displacement of the camera in the X-axis direction, is the displacement of the camera in the Y-axis direction, is the displacement of the camera in the Z-axis direction; the translation vector T is used to translate the point from the world coordinate system to the camera coordinate system, and the calculation formula is ; Through the external parameter matrix The point in the world coordinate system can be Transform the point into the camera coordinate system , the calculation formula is: ; In the formula, the point in the world coordinate system is The constant "1" is added to the formula to form a homogeneous coordinate system, which is used to uniformly describe rotation and translation operations. Through the homogeneous coordinate system, the rotation matrix R and the translation vector T can be combined together, that is, , which represents the entire coordinate transformation.

[0084] Define the shelf area, clarify the spatial range and size parameters of the shelf, and use them as the basis for grid division. The size parameters include the total width of the shelf. , Total Height , Shelf depth (i.e. depth information d); divide the shelf into grid units of uniform size to facilitate the definition of goods and calculation of the center point; define the size of the grid, divide it into n grid units horizontally and m grid units vertically, and regard the depth direction as a fixed value (not divided); calculate the width of the grid unit , calculate the height of the grid cell ; Define the coordinate range of the grid unit, including: horizontal coordinate range , for example, the horizontal range of the i-th grid cell is , ; Vertical coordinate range , for example, the vertical range of the jth grid cell is: , ; Depth coordinate (i.e. the center depth of the shelf surface); calculate the center point coordinates of each grid unit , where the horizontal center coordinates ; Vertical center coordinate ; Depth center coordinates ; Through the above steps, the unique center point coordinates can be calculated for each grid unit of the shelf , as a benchmark for product positioning and management.

[0085] Actual space mapping, the pixel coordinates obtained from the camera Using the depth information d (i.e. the distance from the pixel to the optical center of the camera, which is used to supplement the third-dimensional information of the image midpoint and turn the pixel into a true three-dimensional coordinate point), the camera's intrinsic parameter matrix and the external parameter matrix , mapping it to the world coordinate system , and finally realize the correspondence between the space point and the grid unit on the shelf; the mapping expression is: ; In the formula, The pixel coordinates are constructed into a three-dimensional homogeneous form, that is, the pixel coordinates +Depth information d gets the camera coordinates; internal parameter matrix is the point where the pixel coordinates are extended by the depth information d Convert to a point in the camera coordinate system , is the inverse matrix of the intrinsic parameter matrix, which represents the reverse normalization from the pixel plane to the normalized plane of the camera coordinate system (removing the influence of the image intrinsic parameters); It is the inverse matrix of the external parameter matrix, and its function is to transform the points in the camera coordinate system into the world coordinate system.

[0086] During the camera space calibration process, due to the checkerboard corner detection and depth information d error, the pixel coordinates and depth information d will be affected by noise, the external parameter matrix Drift may occur due to slight movement of the camera or environmental changes (such as temperature), resulting in poor calibration accuracy and dynamic stability of the camera. The Kalman filter model is defined as the optimization model for camera spatial calibration. The optimization model works in the following ways:

[0087] Define the system state vector , is the three-dimensional position of a point in the world coordinate system; the model observation value is defined as , It is the world coordinate calculated by the camera calibration formula (i.e., mapping expression); the prediction model of the camera position and calibration environment in a short period of time is defined as: , in some specific scenarios, when the system state model is static, that is, the system has no obvious movement (for example, the position remains unchanged), the predicted value at the next moment is the updated value at the previous moment; where, Represents the predicted state estimate, that is, time, according to The state estimate at time ) predict the state of the system; represents the updated state estimate, i.e. At time t, the system state is updated by the past measured values ​​and predicted values; using the Kalman gain Fuse the current measured values ​​and predicted values, update the prediction model, ; In the formula, Represents the current observation value, that is, at the current time The actual measured value obtained about the state; Represents the projection of the predicted value in the observation space; is the observation matrix, which is used to estimate the predicted state Mapped to the observation space, ensuring that it is consistent with the current observation value have the same form of expression; The observation residual represents the gap between the observed value and the predicted value, which measures whether the current prediction is consistent with the actual observation; Kalman gain Is a weight matrix used to balance the credibility of the predicted value and the observed value. The calculation formula is: ;in, is the prediction error covariance matrix, reflecting the uncertainty in the prior prediction; is the transpose of the observation matrix, which is used to maintain the consistency of the matrix dimension; is the measurement noise covariance matrix, which represents the sensor noise and the uncertainty of the measurement data; Represents the combined uncertainty between predictions and measurements; is the correction factor used to correct the predicted value; for the prediction error covariance matrix Update, the update formula is: ;in, is the updated error covariance matrix, which represents the uncertainty of the current state estimate produced by the update step;

[0088] is the unit matrix, which is used to maintain dimensional consistency. Each frame of data is input into the optimization model, and the above steps are repeated to continuously optimize the state estimation, so as to obtain the optimization results of the camera internal and external parameter calibration and the optimization results of the midpoint of the world coordinate system, thereby improving the accuracy of the camera space calibration.

[0089] More specifically, in this solution, the data obtained by scanning the goods includes: the category of the goods Identified by target detection module (such as YOLOv8, used to detect the type and quantity of goods in each frame); the location of the rectangular bounding box of the goods , the object detection module provides the rectangular position of the product in the image for locating the product, is the horizontal coordinate value of the leftmost side of the rectangular border. is the vertical coordinate value of the top side of the rectangular border, Represents the total width of the rectangular bounding box, Represents the total height of the rectangular bounding box; the feature description vector of the product , extract the feature description vector (such as color and texture features) of each product through the image processing module, which is used to distinguish or track the products;

[0090] Based on product category , the location of the rectangular bounding box of the product and the product feature description vector Generate the initial status table of goods ,in, is the product depth value, Initialization completion time; product initial status table After completion, it is stored and used for subsequent product management and status comparison testing.

[0091] Furthermore, in this solution, the method of determining whether a product has been taken out, put back or adjusted includes:

[0092] Product removal: If a product is not detected in the current frame image and its position is within a continuous time period If it is empty, it is determined that the product has been taken out; is the time threshold (in seconds), which is determined by the frame rate (FPS) and the number of consecutive undetected frames. The calculation formula is: ,For example, the frame rate of the camera is 50FPS, the time threshold , then the product needs to be not detected in 50 consecutive frames of images before it will be judged as taken out;

[0093] Product return: If a product appears in the current frame image but does not exist in the previous frame image, the product is considered to be returned;

[0094] Position change: If the product position is slightly shifted (i.e. , , is less than the judgment threshold), it is judged as an adjustment behavior; , , ;in, is the horizontal position change, that is, the horizontal movement distance of the product position (in image pixels), is the vertical position change, that is, the distance the product moves in the vertical direction (in image pixels). is the change in depth position, i.e. the distance the product moves in the depth direction (in image pixels); is the upper left corner position of the rectangular bounding box of the product in the previous frame image, is the upper left corner position of the rectangular bounding box of the product in the current frame image, is the depth center coordinate of the rectangular bounding box of the product in the previous frame image, is the depth center coordinate of the rectangular bounding box of the product in the current frame image; the judgment threshold includes the horizontal position change threshold , vertical position change threshold and depth position change threshold (unit is image pixel), set by experiment; the judgment method is , and , it is considered as a product position adjustment, which is used to distinguish between slight adjustments and significant moves of products;

[0095] By outputting the real-time status table of goods, the dynamics of goods can be processed efficiently and accurately, thereby performing precise inventory management of goods.

[0096] Furthermore, in this solution, the methods of capturing user behaviors include:

[0097] Hand key point detection: The camera captures the user's hand key points (such as thumb, palm, wrist, etc.) based on a human posture estimation model (such as OpenPose or Mediapipe), and generates a hand motion path in three-dimensional space to obtain the user's hand trajectory set , represents the movement trajectory of the user's hand during a period of time in the video, and records the continuous movement position data of the key points of the hand over time from the start to the end of detection. It is used to capture the dynamic behavior of the hand (such as whether it interacts with the product) and provide basic input data for subsequent user behavior classification (such as taking out, putting back or adjusting the product); ,in, is the horizontal coordinate of the hand in the image, is the vertical coordinate of the hand in the image, is the depth information of the hand in three-dimensional space, It is the timestamp of the current frame in the entire motion trajectory;

[0098] Interaction detection: based on the user's hand trajectory collection Determine whether the user's hand motion trajectory overlaps with the rectangular boundary box of the product, that is, whether the hand enters the area where the product is located, triggering the product's interactive behavior. The judgment conditions are: and ,in, is the left edge of the rectangular bounding box of the product in the horizontal direction. is the right edge of the rectangular bounding box of the product in the horizontal direction. is the top vertical boundary of the rectangular bounding box of the product. The bottom boundary of the rectangular bounding box of the product in the vertical direction; if the camera detects that the hand interacts with the product, the interaction time is recorded , Product category and where the interaction occurs Based on the dynamic relationship between the hand motion trajectory and the rectangular bounding box of the product, the interaction behavior is divided into two types of interaction events: entry behavior (the hand enters from outside the rectangular bounding box of the product), indicating that the user begins to interact with the product, such as adjusting the product or preparing to take it out. The trigger condition is that in the current frame: or , in the next frame: and ; The move-out behavior (the hand leaves the rectangular bounding box of the product) indicates that the user may have completed the operation on the product, such as taking out the product or stopping adjustment. The triggering condition is in the current frame: and , in the next frame or ;

[0099] Classify user interaction behaviors: Based on interaction events and time series features, classify user interaction behaviors with products, and clarify the user's specific operations (including taking out products, putting back products, and adjusting products); collect user hand trajectories The rectangular bounding box of the product changes dynamically (i.e., the dynamic change of the rectangular bounding box of the product over time, ;in, The timestamp represents the time when the rectangular bounding box of the product changes) is input into the pre-built classification model, and the classification model identifies the user's specific interaction behavior. , respectively:

[0100] Taking out a product is defined as: the user's hand moves out of the rectangular bounding box of the product, and the rectangular bounding box of the product disappears from the scene, indicating that the user may take the product away from the shelf; the triggering conditions are:

[0101] In time : (The hand is within the rectangular bounding box of the product);

[0102] In time : (The hand moves out of the rectangular bounding box of the product);

[0103] And the rectangular bounding box of the product is at time disappear: (The rectangular bounding box of the product disappears), Indicates at time In the image frame, the object detection module does not detect the rectangular bounding box of the product;

[0104] Putting the product back is defined as: the user's hand enters from outside the rectangular bounding box of the product, and a new rectangular bounding box of the product is added to the scene, indicating that the user may put the product back on the shelf; the triggering conditions are:

[0105] In time : (The hand is outside the rectangular bounding box of the product);

[0106] In time : (The hand enters the rectangular bounding box of the product);

[0107] And the rectangular bounding box of the product is at time Added: (The rectangular bounding box of the product is added);

[0108] Adjusting a product is defined as the user's hand moving within the rectangular bounding box of the product, causing the product's position to change slightly, indicating that the user may be adjusting the product's position or viewing the product. The triggering conditions are:

[0109] In time , : and ;

[0110] And the change in product position , and Less than the judgment threshold , and ;

[0111] The classification model generates feature data based on each specific interaction behavior of the user. ;in, It is The interaction between the user and the product is detected (recording an event or behavior), which is used to describe the key information of each interaction event; is the type of interaction behavior;

[0112] Feature data generated by the classification model , can track user interaction records, dynamically update the real-time status table of goods in the vending machine system, and provide accurate guarantee for the inventory management of goods.

[0113] It is worth mentioning that in this case, there are many cases of product identification errors, namely:

[0114] Fast motion capture error: The user's fast operation may cause the camera to lose some key data between frames when capturing the image, resulting in misjudgment or multiple counting. For example, if a user quickly takes out a product and puts it back, it may be misjudged as two independent "pick and put" operations;

[0115] Multiple pick-and-place recognition errors: Users may repeatedly pick up and put down items from the same location, causing the system to misjudge these as multiple independent interactions. For example, if a user repeatedly picks up an item, examines it, and then puts it back, the system may count it multiple times.

[0116] Recognition error caused by incomplete removal of the product: If the user does not completely remove the product, the system may miss the removal behavior, for example, part of the product bounding box is still in the camera's field of view;

[0117] Error in identifying changes in product placement: Users may make slight adjustments to the product position (such as rotating or moving it), but the product is not taken out or put back, and the system may mistakenly identify it as a take-and-place action;

[0118] By passing the feature data Input into the pre-built error correction model to obtain the corrected feature data ; Then locate the source of error in product identification and improve the accuracy of product identification; The construction method of the error correction model includes:

[0119] The correction method for fast motion capture error is to introduce a defined parameter, the time window , for example, 1 second, is used to determine whether multiple repeated operations occur in a short period of time; in this time window Feature data for all frames Perform statistics and corrections, check the time intervals of these frames Is it less than the time window? , ; If the time window If multiple operations occur at the same product location within a time window and the product status information does not change significantly, these operations are considered to belong to the same valid interaction. Multiple operations within are merged into one valid operation;

[0120] The correction method for multiple pick-and-place recognition errors is to check each interaction behavior After the product status information is obtained, if the same product enters or leaves the bounding box multiple times, but its product status information does not actually change, these repeated operations are filtered out and only one valid operation is counted;

[0121] The product has not completely moved out of the recognition error: Check the center point of the rectangular bounding box of the product The offset and with a pre-set offset threshold For comparison, if , and the rectangular bounding box of the product disappears in the next frame image, it is determined that the product is actually taken out; if the above conditions are met, but the camera does not detect the removal behavior, then an additional removal operation is recorded; offset The calculation formula is: ;

[0122] The correction method for the recognition error of the change in the product placement position is: detect the offset of the center point of the rectangular bounding box of the product and with a pre-set offset threshold Compare and then check the product status , the product status in the current frame image Determine the product status in the previous frame image Compare to determine whether the product has not changed significantly; if and , it is judged as an adjustment behavior; that is, the quantity statistics of the goods are corrected, that is, the quantity after correction is equal to the quantity before correction;

[0123] Based on the above, the expression of the error correction model is constructed:

[0124] ;

[0125] In the formula, Is a conditional statement used to determine each feature data Should it be included in the result set, i.e. correct the feature data Inside; For feature data Total number of;

[0126] The error correction model allows the camera to more accurately distinguish between slight adjustments to products, repeated operations, and significant pick-and-place behaviors, reducing misjudgments caused by slight product movements (such as placement adjustments) or rapid repeated operations. At the same time, it can supplement records based on changes in time, location, and status to avoid omissions, and improve the detection accuracy of product image recognition.

[0127] It is worth noting that in this case, based on the modified feature data By building a user behavior prediction model, we can obtain the time dependency and behavior pattern of user behavior, improve the ability of the vending machine warehouse management system to understand user behavior, and optimize the inventory management of goods; accurately classify and predict user behavior, reduce misjudgment and omissions in product image recognition, and then optimize the response speed and resource allocation of vending machines, improve user experience and operational efficiency; ways to build a user behavior prediction model include:

[0128] Continuously modify feature data for each user behavior Constructed as time series data, the shape is ;in, is the number of time steps, For each interaction The number of features (such as time, product location, product status) is normalized to make the range of numerical features (such as time interval, location coordinates) within Between; Based on interactive behavior Label each time series data for supervised learning;

[0129] Define the LSTM network structure as a user behavior prediction model and transform the time series data Input into the user behavior prediction model to obtain the category probability distribution at each time step ; This enables the user behavior prediction model to classify time series data and optimize the product layout and warehouse management of the vending machine based on the analysis of user behavior patterns. For example, it can predict which products will be in high demand in a specific time period and replenish them in advance.

[0130] The LSTM network structure includes an embedding layer, an LSTM layer, a fully connected layer, and a Softmax layer;

[0131] The embedding layer receives input time series data , the time series data The features at each time step Mapped to a high-dimensional feature space, ; In the formula, is the embedding representation, indicating the High-dimensional feature vector for each time step; is the input feature, indicating The original feature vector of time steps; is the embedding weight matrix, which is used to map the original features into a high-dimensional space. The shape is (generally ), is the feature dimension after embedding; is the embedding bias vector, which is used to adjust the result of feature mapping; embedding representation is the output of the embedding layer, with a shape of ;

[0132] The LSTM layer receives the output embedding representation of the embedding layer ,LSTM layer includes input gate, forget gate, candidate memory, memory update, output gate and hidden state;

[0133] The input gate embeds the representation As the input feature, we get the output vector of the input gate , ; Where the output vector of the input gate is Indicates the current time step The input features (i.e., the embedding representation ) for the current memory state the extent of the impact; is the activation function, which is used to convert the output vector The value is limited to 0 means to ignore the embedding representation completely. , 1 means fully accepting the embedded representation ; is the weight matrix of the input gate, which is used to convert the current time step The input features are converted into a feature space suitable for the input gate, with a shape of , is the dimension of the hidden state, is the dimension of the input feature; is the weight matrix of the hidden state of the previous time step, which is used to transform the hidden state of the previous time step into the current time step The feature space of the input gate has the shape ; is the hidden state of the previous time step, including the embedded representation With the global information at the previous time step, the shape is ; is the bias vector of the input gate, which is used to translate the calculation result of the input gate and has the shape of ;

[0134] The forget gate determines the memory state of the previous time step How much information needs to be retained to the current time step; embedding and the hidden state at the previous time step As the input feature, we get the output vector of the forget gate , ; Where, the output vector of the forget gate is The value of Range, used to extend the memory state of the previous time step Whether each part is retained or forgotten, 0 means completely forgotten, 1 means completely retained; is the weight matrix of the forget gate, which is used to convert the current time step The input features are converted into the feature space of the forget gate, with the shape of ; is the weight matrix of the hidden state of the previous time step, which is used to transform the hidden state of the previous time step Transformed into a feature space suitable for the forget gate, the shape is ; is the bias vector of the forget gate, which is used to translate the calculation result of the forget gate and has the shape of ;

[0135] Candidate memory is the new information generated at the current time step, which is used to update the memory state , represents the potential memory content of the current time step, providing a new source of information for memory updating; embedding and the hidden state at the previous time step As input features, we get candidate memory vectors , ; Where, the candidate memory vector Indicates the current time step New information is used to update the memory state ; is the hyperbolic tangent activation function, which is used to transform the candidate memory vector The value is compressed to In the range, -1 means completely suppressing the information, and 1 means completely activating the information; is the weight matrix of the candidate memory, which is used to convert the current time step The input features are converted into the feature space of the adaptive candidate memory, with the shape of ; is the weight matrix of the hidden state of the previous time step, which is used to transform the hidden state of the previous time step Transformed into the feature space of the candidate memory, the shape is ; is the bias vector of the candidate memory, which is used to translate the calculation results of the candidate memory and has the shape of ;

[0136] Memory Update is used to update the memory state ; The memory state of the previous time step , candidate memory vector , the output vector of the input gate And the output vector of the forget gate As input features, get the memory state of the current time step ; Where, the memory state of the current time step is Stores long-term information of time series data; It is the symbol of bit-by-bit multiplication of elements, indicating that the elements at corresponding positions in the matrix are multiplied bit-by-bit;

[0137] The output gate is used to control the memory state of the current time step How much information can be obtained through hidden state Output, embedding representation and the hidden state at the previous time step As the input feature, we get the output vector of the output gate , ; Where, the output vector of the output gate is The value of In the range, it indicates the memory state of the current time step How much information can be obtained through hidden state Output, 0 means completely ignored, 1 means completely accepted; is the weight matrix of the output gate, which is used to convert the current time step The input features are converted into the output gate feature space, the shape is ; is the weight matrix of the hidden state of the previous time step, which is used to transform the hidden state of the previous time step Transformed into a feature space suitable for the output gate, the shape is ; is the bias vector of the output gate, which is used to translate the calculation result of the output gate and has the shape of ;

[0138] The hidden state is the final output of the LSTM layer, which represents the feature expression of the current time step and the memory state of the time step And the output vector of the output gate As input features, we get the hidden state , ;

[0139] The LSTM network structure outputs the hidden state at each time step , the shape is ,in, Is a hidden state The dimension of hidden state Input to the fully connected layer, the hidden state of the fully connected layer for each time step Perform linear transformation to obtain the classification vector , ; In the formula, It is used to map the hidden state The weight matrix determines the hidden state How the information is converted into a classification vector , the shape is ,in for Dimensions; is the fully connected layer bias vector, used for conditional linear change results , the shape is ; The output of the fully connected layer is the classification vector , the shape is ,in is the number of categories;

[0140] The classification vector Input into the Softmax layer, the Softmax layer classifies the vector Converted into probability distribution for classification; for each time step The classification vector , calculate the probability of each category, the calculation formula is ; In the formula, is the input score, indicating the time step When, category A score (classification vector is a dimension of The vector of Categories The scores of each category Corresponding to an input score ); is an exponential function used to input scores Perform nonlinear mapping and input scores Mapping to positive range ; is the normalization factor used to normalize all categories Input score Normalized to probability value; Represents the category probability distribution of each time step, that is, the corresponding time step When, category The probability of is .

[0141] Collect historical correction feature data , use it as the training input set of the user behavior prediction model, and set the number of iterations; then initialize the network parameters of the user behavior prediction model, including: the weight matrix of the input gate , the bias vector of the input gate , the weight matrix of the forget gate , the bias vector of the forget gate , the weight matrix of candidate memory , the bias vector of the candidate memory , the weight matrix of the output gate , the bias vector of the output gate ;

[0142] Define the cross entropy loss function of the user behavior prediction model to measure the distance between the model prediction result and the true label. Its expression is: ; In the formula, is the value of the cross entropy loss function; is the number of samples, i.e. time series data the number of is the true label, i.e. Samples in category The true class probability distribution ; is the probability predicted by the model, that is, the model predicts the Samples in category The predicted class probability distribution ; is the predicted category probability distribution The logarithm is taken to widen the gap between different prediction values ​​and increase the penalty of the model for low probability predictions. Close to 1, the logarithm is close to 0, the loss is small, and when Close to 0, the logarithm tends to negative infinity, and the loss is large; For Category the number of

[0143] The training input set is input into the user behavior prediction model, followed by the embedding layer, LSTM layer, fully connected layer and Softmax layer; the corresponding cross entropy loss function value is calculated, and the error gradient is back-propagated from the output layer to the input layer; according to the error gradient, the network parameters are updated using an optimization algorithm (such as gradient descent, Adam, etc.) to reduce the value of the loss function; the sequence data is repeatedly trained until the user behavior prediction model reaches a preset number of iterations; this indicates that the user behavior prediction model converges, which can be understood as the value of the loss function no longer changes, and thus indicates that the user behavior prediction model can accurately predict the time dependency and behavior pattern of user behavior.

[0144] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements will not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for intelligently identifying commodity images in vending machines, characterized in that: The following steps are involved: Step 1: Install a camera inside the vending machine and perform spatial calibration on the camera; Step 2: The camera scans the goods on the shelf and generates an initial state table of the goods; Step 3: The camera detects the status of the goods in real time and obtains the product image, extracts the status information of all the goods in each frame image, compares the status information of the goods in the current frame image with the status information of the goods in the previous frame image, determines whether any goods are taken out, put back or adjusted, and outputs the real-time status table of the goods; The trigger conditions for taking out goods are: In time : ; In time : ; And the rectangular bounding box of the product is at time disappear: , Indicates at time In the image frame, the object detection module does not detect the rectangular bounding box of the product; The trigger conditions for returning the product are: In time : ; In time : ; And the rectangular bounding box of the product is at time Added: ; The trigger conditions for adjusting products are: In time , : and ; And the change in product position includes , and Less than the judgment threshold , and ;in, The rectangular bounding box of the product changes dynamically. is the user's hand trajectory collection; Step 4: The camera captures the user's interactive behavior and inputs it into the classification model to obtain feature data , used to dynamically update the real-time status table of commodities; Step 5: Feature data Input into the error correction model to obtain the corrected characteristic data , which is used to eliminate the error in product image recognition; the error correction model is constructed as follows: The definition of fast motion capture error is that when the user's quick operation causes the camera to capture the picture, some key data is lost between frames, resulting in misjudgment or multiple counting. The correction method of fast motion capture error is defined as: Feature data for all frames Perform statistics and corrections, check frame time intervals Is it less than the time window? ; If the time window If multiple operations occur at the same product location within a time window and the product status information does not change significantly, the operations are considered to belong to the same valid interaction. Multiple operations within are merged into one valid operation; Define multiple pick-and-place recognition errors as users repeatedly picking up and placing items from the same location, which causes the system to misjudge them as multiple independent interactions. Define the correction method for multiple pick-and-place recognition errors as: check each interaction behavior After the product status information is entered, if the same product enters or leaves the bounding box multiple times, but its product status information does not actually change, it is only counted as one valid operation; Define that if the user does not completely remove the product, the system will miss the removal behavior as the product not completely removed recognition error; define that the correction method for the product not completely removed recognition error is: check the offset of the center point of the rectangular bounding box of the product and with a pre-set offset threshold For comparison, if , and the rectangular bounding box of the product disappears in the next frame of the image, it is determined that the product is actually taken out; if the above conditions are met, but the camera does not detect the removal behavior, then an additional removal operation is recorded; Define that the user makes a slight adjustment to the position of the product, but the product is not taken out or put back, and the system misjudges the pick-and-place behavior as a product placement change recognition error; define the correction method for the product placement change recognition error as: detect the offset of the center point of the rectangular bounding box of the product and with a pre-set offset threshold Compare and then check the product status , the product status in the current frame image Determine the product status in the previous frame image Compare to determine whether the product has not changed significantly; if and , it is judged as adjustment behavior; Construct the expression of the error correction model: ； In the formula, Is a conditional statement used to determine each feature data Should it be included in the correction feature data? Inside; is the time interval, ; is the offset of the center point of the rectangular bounding box of the product, , is the upper left corner position of the rectangular bounding box of the product in the previous frame image, is the upper left corner position of the rectangular bounding box of the product in the current frame image, is the depth center coordinate of the rectangular bounding box of the product in the previous frame image, The depth center coordinates of the rectangular bounding box of the product in the current frame image; is the offset threshold; is the product status of the current frame image, The product status of the previous frame image; For feature data Total number of; Step 6: Correct the feature data Input into the user behavior prediction model to obtain the time dependency and behavior pattern of user behavior.

2. The method for intelligently identifying commodity images in vending machines according to claim 1, characterized in that: The method of spatially calibrating the camera in step 1 includes: Calibrate the camera to obtain the camera's intrinsic parameter matrix and the external parameter matrix ; Define the spatial scope and size parameters of the shelf, including: the total width of the shelf , Total Height , Shelf depth ; Divide the shelf into grid units of uniform size and obtain the center point coordinates of the grid units ; Based on the size of the shelf, the internal parameter matrix and the external parameter matrix The pixel coordinates on the image acquired by the camera Map it to the world coordinate system to get the three-dimensional position of the midpoint in the world coordinate system ; The 3D position of the point in the world coordinate system Input into the optimization model to obtain the optimization results of the camera internal and external parameter calibration and the optimization results of the midpoint of the world coordinate system.

3. The method for intelligently identifying commodity images in vending machines according to claim 2, characterized in that: The optimization method of the optimization model is: Define the system state vector and the model observations are ; Define the prediction model of camera position and calibration environment in a short static state: ; In the formula, represents the predicted state estimate; represents the updated state estimate; Using Kalman Gain Fuse the current measured values ​​and predicted values, update the prediction model, ; In the formula, Represents the current observation value; Represents the projection of the predicted value in the observation space; is the observation matrix; is the observation residual; Kalman gain Is a weight matrix, calculated as: ;in, is the prediction error covariance matrix; is the transpose of the observation matrix; Represents the combined uncertainty between predictions and measurements; is the correction amount; is the measurement noise covariance matrix; Forecast error covariance matrix Update, the update formula is: ;in, is the updated error covariance matrix; is the identity matrix; Input each frame of data into the optimization model and repeat the above steps.

4. The method for intelligently identifying commodity images in vending machines according to claim 1, characterized in that: The method of obtaining product data in step 2 includes: Identify the category of goods through the object detection module and the location of the rectangular bounding box of the product , is the horizontal coordinate value of the leftmost side of the rectangular border. is the vertical coordinate value of the top side of the rectangular border, Represents the total width of the rectangular bounding box, Represents the total height of the rectangular bounding box; extracts the feature description vector of each product through the image processing module ; Product initial status table ;in, is the product depth value, Initialization completion time.

5. The method for intelligently identifying commodity images in vending machines according to claim 1, characterized in that: The method for determining whether a product has been taken out, put back or adjusted in step 3 includes: If a product is not detected in the current frame image and its position is within a continuous time period If it is empty, it is determined that the product has been taken out; is the time threshold, determined by the frame rate and the number of consecutive undetected frames Calculated; if a product appears in the current frame image, but does not exist in the previous frame image, it is determined that the product is put back; if the position of the product is slightly offset, that is , and , it is judged as a product position adjustment; , , ;in, is the horizontal position change, is the vertical position change, is the depth position change; the judgment threshold includes the horizontal position change threshold , vertical position change threshold and depth position change threshold .

6. A method for intelligently identifying commodity images in vending machines according to claim 5, characterized in that: The method of capturing the user's interactive behavior in step 4 includes: The camera captures the user's hand key points based on the human posture estimation model and obtains the user's hand trajectory set , ,in, is the horizontal coordinate of the hand in the image, is the vertical coordinate of the hand in the image, is the depth information of the hand in three-dimensional space, It is the timestamp of the current frame in the entire motion trajectory; Based on the user's hand trajectory collection Determine whether the user's hand motion trajectory overlaps with the rectangular boundary box of the product to trigger the product's interactive behavior. The judgment conditions are: and ,in, is the left edge of the rectangular bounding box of the product in the horizontal direction. is the right edge of the rectangular bounding box of the product in the horizontal direction. is the top vertical boundary of the rectangular bounding box of the product. The bottom boundary of the rectangular bounding box of the product in the vertical direction; If the camera detects that the hand interacts with the product, the interaction time is recorded , Product category and where the interaction occurs Based on the dynamic relationship between the hand motion trajectory and the rectangular bounding box of the product, the interaction behavior is divided into two types of interaction events: Enter the behavior, the trigger condition is in the current frame: or , in the next frame: and ; Remove behavior, the trigger condition is in the current frame: and , in the next frame or ; Based on interaction events and time series features, the user's hand trajectory is collected The rectangular bounding box of the product changes dynamically Input into the pre-built classification model to identify the user's specific interaction behavior , and generate feature data based on each specific interaction behavior of the user ;in, is the timestamp, Indicates The feature data of the user's interaction with the product is detected. is the type of interactive behavior; interactive behavior Includes: taking out goods, putting goods back, and adjusting goods.

7. The method for intelligently identifying commodity images in vending machines according to claim 1, characterized in that: The method of constructing the user behavior prediction model in step 6 includes: Continuously modify feature data for each user behavior Constructed as time series data, the shape is ;in, is the number of time steps, For each interaction The number of features; normalize the numerical features; Define the LSTM network structure as a user behavior prediction model and transform the time series data Input into the user behavior prediction model to obtain the category probability distribution at each time step ; Collect historical correction feature data , use it as the training input set of the user behavior prediction model, and set the number of iterations; initialize the network parameters of the user behavior prediction model, including: the weight matrix of the input gate , the bias vector of the input gate , the weight matrix of the forget gate , the bias vector of the forget gate , the weight matrix of candidate memory , the bias vector of the candidate memory , the weight matrix of the output gate , the bias vector of the output gate ; Define the cross entropy loss function of the user behavior prediction model, which is expressed as: ; In the formula, is the value of the cross entropy loss function; is the sample size; is the true label; The probability predicted by the model; is the predicted category probability distribution The logarithm of For Category Quantity; The training input set is input into the user behavior prediction model, followed by the embedding layer, LSTM layer, fully connected layer and Softmax layer; the corresponding cross entropy loss function value is calculated, and the error gradient is back-propagated from the output layer to the input layer; according to the error gradient, the Adam optimization algorithm is used to update the network parameters to reduce the value of the loss function; the time series data is repeatedly trained until the user behavior prediction model reaches the preset number of iterations to obtain the time dependency and behavior pattern of the user behavior.

8. The method for intelligently identifying commodity images in vending machines according to claim 7, characterized in that: The LSTM network structure includes an embedding layer, an LSTM layer, a fully connected layer and a Softmax layer; the embedding layer receives input time series data , the time series data The features at each time step Mapped to high-dimensional feature space to obtain embedded representation ; In the formula, is the embedding representation, indicating the High-dimensional feature vector for each time step; is the input feature, indicating the The original feature vector of time steps; is the embedding weight matrix; is the embedding bias vector; The LSTM layer receives the output embedding representation of the embedding layer , get the hidden state ; Hide the state Input to the fully connected layer, the hidden state of the fully connected layer for each time step Perform linear transformation to obtain the classification vector ; In the formula, It is used to map the hidden state The weight matrix of is the bias vector of the fully connected layer; The classification vector Input to the Softmax layer, the Softmax layer performs The classification vector Calculate each category The probability is calculated as ; In the formula, is the input score, indicating the time step When, category A score of is an exponential function used to input scores Perform nonlinear mapping and input scores Mapping to positive range ; is the normalization factor used to normalize all categories Input score Normalized to probability value; Represents the class probability distribution at each time step.

Images