Interaction behavior identification method and system for intelligent unmanned selling cabinet

Abstract

The invention discloses an interactive behavior recognition method for an intelligent unmanned selling cabinet, and relates to the field of behavior recognition, and the method comprises the steps: detecting a relative distance between a target customer and the intelligent unmanned selling cabinet; when the relative distance is smaller than a preset distance threshold value, obtaining a customer dynamic video of the target customer; analyzing a first abnormal behavior of the target customer according to the customer dynamic video; the operation state of the intelligent unmanned selling cabinet is adjusted according to the first abnormal behavior analysis result; acquiring sensor information of the intelligent unmanned selling cabinet; and identifying a second abnormal behavior of the target customer according to the sensor information. According to the invention, all-weather, high-precision and high-efficiency identification and monitoring can be effectively carried out on the interaction behavior of the intelligent unmanned selling cabinet.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of behavior recognition, and more particularly to an interactive behavior recognition method and system for an intelligent unmanned vending machine. Background Art

[0002] With the rapid development of modern society, unmanned vending machines are gradually developing in the direction of intelligence. Today's intelligent unmanned vending machines have the advantages of 24-hour service, various payment methods, and convenient pickup. It is precisely because of these advantages that the distribution range of intelligent unmanned vending machines is becoming wider and wider, and the number of users is gradually increasing. However, with the wide distribution of intelligent unmanned vending machines and the increase in the number of users, more and more problems are encountered, such as customers maliciously damaging intelligent unmanned vending machines, maliciously replacing goods, and other abnormal interactive behaviors.

[0003] The existing technology mainly relies on manual recognition to identify these abnormal interactive behaviors. It manually determines whether the target customer has the above abnormal interactive behaviors. However, manual recognition is inefficient and cannot perform behavioral recognition and analysis 24 hours a day. In addition, for the widely distributed smart unmanned vending machines, the manpower cost required is also very high, which will cause a waste of human resources. Summary of the invention

[0004] The embodiments of the present application provide a method and system for identifying interactive behaviors of smart unmanned vending machines, which are used to solve the problem that the existing technology is unable to identify and monitor the interactive behaviors of smart unmanned vending machines around the clock, with high precision and high efficiency.

[0005] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

[0006] In a first aspect, a method for identifying interactive behaviors for an intelligent unmanned vending machine is provided, the method comprising:

[0007] Detecting the relative distance between the target customer and the intelligent unmanned vending cabinet by means of an infrared sensor disposed outside the intelligent unmanned vending cabinet;

[0008] When the relative distance is less than a preset distance threshold, a customer dynamic video including the target customer is acquired by an image acquisition device disposed outside the intelligent unmanned vending cabinet;

[0009] Performing abnormal behavior analysis on the customer dynamic video, and determining whether the target customer has a first abnormal behavior according to the abnormal behavior analysis result;

[0010] If the target customer has the first abnormal behavior, the smart unmanned vending machine is adjusted from a normal operating state to a cabinet door locked state within a preset time, and the cabinet door cannot be unlocked in the cabinet door locked state;

[0011] If the target customer does not have the first abnormal behavior, maintaining the normal operating state of the smart unmanned vending machine;

[0012] The sensor information of the smart unmanned vending cabinet within the information collection time period is obtained by a sensor disposed inside the smart unmanned vending cabinet, wherein the sensor information includes gravity sensor information and image sensor information, the starting point of the information collection time period is the initial time node when the target customer completes unlocking the cabinet door, and the ending point of the information collection time period is the deadline time node when the smart unmanned vending cabinet completes closing the cabinet door;

[0013] The gravity sensor information and the image sensor information are combined to identify a second abnormal behavior of the target customer within the information collection time period.

[0014] Optionally, performing abnormal behavior analysis on the customer dynamic video and determining whether the target customer has a first abnormal behavior according to the abnormal behavior analysis result comprises the following steps:

[0015] Inputting the customer dynamic video into a pre-built visual saliency model, performing saliency analysis on the customer dynamic video through the visual saliency model, and obtaining behavior saliency values ​​of all behavior pixels in each frame of the customer behavior image in the customer dynamic video;

[0016] For any frame of the customer behavior image, the behavior pixel points whose behavior significance value in the customer behavior image is greater than a preset significance value threshold are screened out to obtain a plurality of significant pixel points;

[0017] Integrate all the significant pixel points to obtain the character area of ​​the customer behavior image;

[0018] Using a dynamic behavior filter to perform convolution analysis on the customer behavior image where the character area exists, to obtain a behavior response map of the customer behavior image, and to calculate the character interest points of the customer behavior image based on the behavior response map;

[0019] Calculating the local gradient of all interest point neighborhoods of the customer behavior image using a local feature extraction algorithm, wherein the interest point neighborhood is an image area within a preset range around the person's interest point;

[0020] Calculate the local feature descriptors of all the neighborhoods of the interest points according to the local gradients, and integrate all the local feature descriptors to obtain the local behavior features of each of the customer behavior images;

[0021] Calculating the optical flow histograms of all the customer behavior images using an optical flow estimation algorithm, and integrating all the optical flow histograms to obtain the global behavior characteristics of the target customer;

[0022] Fusing the local behavior feature corresponding to the customer behavior image with the corresponding global behavior feature to obtain the character behavior feature of the target customer;

[0023] The character behavior feature is input into a pre-built abnormal behavior recognition model to perform abnormal behavior recognition to obtain an abnormal output value, and it is determined whether the target customer has a first abnormal behavior according to the abnormal output value.

[0024] Optionally, the step of performing convolution analysis on the customer behavior image in the character area using a dynamic behavior filter to obtain a behavior response graph of the customer behavior image, and calculating the character interest points of the customer behavior image based on the behavior response graph comprises the following steps:

[0025] Inputting the customer dynamic video with the character area marked into the dynamic behavior filter for filtering;

[0026] Performing convolution processing in different directions on each frame of the customer behavior image of the customer dynamic video that has completed the filtering processing, to obtain a horizontal response map and a vertical response map of each frame of the customer behavior image;

[0027] Combining the horizontal response graphs and the vertical response graphs corresponding to all the customer behavior images to obtain behavior response graphs for all the customer behavior images;

[0028] Calculate the behavior response values ​​of all behavior pixels in the behavior response graph using the response function;

[0029] For each frame of the customer behavior image, respectively calculating the sum of the behavior saliency values ​​of all the behavior pixels in the person area and non-person area in the customer behavior image to obtain a regional saliency value and a non-regional saliency value, wherein the non-person area is an area other than the person area in the customer behavior image;

[0030] According to all the region saliency values ​​and the non-region saliency values, weights are respectively assigned to the human region and the non-human region;

[0031] The pixel threshold formula is used to assign thresholds to the behavior pixels in the character area and the non-character area after weight assignment to obtain the regional response threshold and the non-regional response threshold. The pixel threshold formula is as follows:

[0032]

[0033]

[0034] Among them, D a is the regional response threshold, D b is the non-region response threshold, is the regional saliency value of the customer behavior image corresponding to the i-th time node, is the non-regional saliency value of the customer behavior image corresponding to the i-th time node, ∈ is a preset fixed value, γ is a preset time period, β a and β b are the weights of the character area and the non-character area, t is the time coordinate of the customer behavior image corresponding to the behavior response graph, G a is the sum of all the regional saliency values ​​of all the customer behavior images, G b is the sum of all the non-regional saliency values ​​of all the customer behavior images;

[0035] For each of the character regions in all the customer behavior images, the behavior pixel points in the character region whose behavior response value is greater than the region response threshold are screened out as character interest points of the customer behavior image;

[0036] The behavior pixel points in the non-person area whose behavior response values ​​are greater than the non-area response threshold are screened out as the person interest points of the customer behavior image.

[0037] Optionally, the response function is as follows:

[0038]

[0039] Wherein, H(x, y) is a two-dimensional Gaussian smoothing kernel, (x, y) is the pixel coordinate of the behavior pixel point on the behavior response map, σ, ω and τ are preset parameters of the dynamic behavior filter, and T is the behavior response map.

[0040] Optionally, the step of calculating the optical flow histograms of all the customer behavior images by using an optical flow estimation algorithm and integrating all the optical flow histograms to obtain the global behavior features of the target customer comprises the following steps:

[0041] Extracting multiple frames of continuous customer behavior images from the customer dynamic video in time order as candidate behavior images;

[0042] Calculate the optical flow vectors of the character interest points in all the candidate behavior images using an optical flow estimation algorithm;

[0043] Calculate the optical flow amplitude and optical flow direction of all the character's interest points according to the optical flow vector of the character's interest point;

[0044] constructing an optical flow histogram for all the candidate behavior images according to the optical flow amplitude and the optical flow direction;

[0045] After traversing all the customer behavior images in the customer dynamic video, all the optical flow histograms are integrated to obtain the global behavior characteristics of the target customer.

[0046] Optionally, the identifying a second abnormal behavior of the target customer within the information collection time period by combining the gravity sensor information and the image sensor information comprises the following steps:

[0047] Extracting the gravity change characteristics of the gravity sensor information at the cut-off time node, and analyzing the gravity change trend of the intelligent unmanned vending machine according to the gravity change characteristics;

[0048] If the gravity change trend is that gravity continues to increase, it is determined that the target customer has a second abnormal behavior during the information collection time period;

[0049] If the gravity change trend is that the gravity continues to decrease, it is determined that the target customer does not have the second abnormal behavior during the information collection time period;

[0050] If the gravity change trend is that the gravity fluctuates up and down, the gravity difference between the initial time node and the end time node is calculated, and the second abnormal behavior of the target customer during the information collection time period is identified by combining the gravity difference and the image sensor information.

[0051] Optionally, the calculating the gravity difference between the initial time node and the cut-off time node, and identifying the second abnormal behavior of the target customer within the information collection time period by combining the gravity difference and the image sensor information comprises the following steps:

[0052] Acquire the initial gravity of the initial time node and the cutoff gravity of the cutoff time node according to the gravity sensor information;

[0053] determining whether a gravity difference value obtained by subtracting the initial gravity from the cut-off gravity is zero, and if the gravity difference value is zero, identifying the second abnormal behavior of the target customer during the information collection time period through the image sensor information;

[0054] If the gravity difference is not zero, analyzing whether the gravity difference is greater than zero; if the gravity difference is less than zero, identifying the second abnormal behavior of the target customer during the information collection time period through the image sensor information;

[0055] If the gravity difference is greater than zero, it is determined that the target customer has the second abnormal behavior during the information collection time period.

[0056] Optionally, the identifying the second abnormal behavior of the target customer within the information collection time period by using the image sensor information comprises the following steps:

[0057] Using the image sensor information collected at the initial time node as the reference image information, and using the image sensor information collected at the cutoff time node as the target image information;

[0058] Performing image registration on the reference image information and the target image information using an image registration technology;

[0059] Extracting a color histogram of the reference image information and the target image information after image registration is completed;

[0060] grayscale processing is performed on the reference image information and the target image information that have completed the registration;

[0061] Using an edge detection algorithm to respectively extract the contours of the commodity in the reference image information and the target image information that have been grayed;

[0062] Acquire shape features of the reference image information and the target image information according to the product outline;

[0063] comparing the color histogram and the shape feature between the reference image information and the target image information respectively, and if the color histogram and the shape feature are the same, determining that the target customer does not have the second abnormal behavior during the information collection time period;

[0064] If the color histogram and / or the shape feature are different, it is determined that the target customer has the second abnormal behavior during the information collection time period.

[0065] Optionally, the method further includes:

[0066] If the target customer does not have the second abnormal behavior, the reference image information and the target image information are respectively input into a preset commodity recognition model to obtain reference commodity information and target commodity information;

[0067] Comparing the reference commodity information with the target commodity information to obtain the types and quantities of commodities sold by the intelligent unmanned vending machine;

[0068] Calculating the weight of the sold goods according to the types of the sold goods and the quantity of the sold goods;

[0069] comparing the absolute value of the difference between the weight of the sold commodity and the gravity difference to see whether they are the same; if the weight of the sold commodity is not the same as the absolute value of the difference, inputting the reference commodity information and the target commodity information into the commodity recognition model again until the weight of the sold commodity is the same as the absolute value of the difference;

[0070] If the weight of the sold goods is the same as the absolute value of the difference, outputting the bill information of the target customer according to the type of the sold goods and the quantity of the sold goods;

[0071] If the target customer has the second abnormal behavior, the intelligent unmanned vending machine is controlled to output an alarm message.

[0072] In a second aspect, the present application provides an interactive behavior recognition system for an intelligent unmanned vending machine, characterized in that it includes:

[0073] a memory configured to store instructions; and

[0074] The processor is configured to call the instructions from the memory and implement the method for interactive behavior identification of an intelligent unmanned vending machine described in any one of the first aspects when executing the instructions.

[0075] Through the above technical solution, the dynamic video of the target customer is obtained, and the first abnormal behavior of the target customer is identified through the dynamic video. If the target customer is identified to have the first abnormal behavior, the smart unmanned vending cabinet is adjusted to the cabinet door locked state to prevent the target customer with the first abnormal behavior from opening the cabinet door and performing further abnormal behavior on the smart unmanned vending cabinet, thereby reducing unnecessary risks for the smart unmanned vending cabinet and extending the service life of the smart unmanned vending cabinet. If the recognition result shows that the target customer does not have the first abnormal behavior, the normal operation state of the smart unmanned vending cabinet is maintained, and the target customer can unlock the smart unmanned vending cabinet. In addition, the sensor information detected by the smart unmanned vending cabinet itself can identify whether the target customer has the second abnormal behavior. If the target customer does not have the second abnormal behavior, the bill information is sent to the target customer. If the target customer has the second abnormal behavior, the alarm information is output in time to ensure the shopping experience of other target customers. Through the above technical solution, a method for accurately and efficiently identifying the interactive behavior of the smart unmanned vending cabinet is provided around the clock, which effectively guarantees the shopping experience of the target customer and extends the service life of the smart unmanned vending cabinet.

[0076] Other features and advantages of the embodiments of the present application will be described in detail in the subsequent specific implementation section. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1A flow chart of an interactive behavior recognition method for an intelligent unmanned vending machine provided in an embodiment of the present application;

[0078] Figure 2 A flowchart for identifying a second abnormal behavior is provided in an embodiment of the present application. DETAILED DESCRIPTION

[0079] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the specific implementation methods described herein are only used to illustrate and explain the embodiments of the present application, and are not used to limit the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0080] It should be noted that if the embodiments of the present application involve directional indications (such as up, down, left, right, front, back...), the directional indications are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0081] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present application, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0082] Figure 1 The following schematically shows a flow chart of a method for identifying interactive behaviors of an intelligent unmanned vending machine according to an embodiment of the present application. Figure 1 As shown, the embodiment of the present application provides an interactive behavior recognition method for an intelligent unmanned vending machine, which may include the following steps:

[0083] S101. Detect the relative distance between the target customer and the smart unmanned vending cabinet by using an infrared sensor disposed outside the smart unmanned vending cabinet.

[0084] In this embodiment, the infrared sensor installed outside the smart unmanned vending machine remains turned on. When a target customer approaches the smart unmanned vending machine, the infrared light emitted by the infrared sensor will be reflected back when it encounters the target customer and received by the infrared sensor. By measuring the time difference from the emission to the reception of the infrared light and combining it with the speed of light, the relative distance between the target customer and the smart unmanned vending machine can be calculated.

[0085] S102: When the relative distance is less than a preset distance threshold, a customer dynamic video including the target customer is obtained by an image acquisition device disposed outside the intelligent unmanned vending cabinet.

[0086] In this embodiment, when the relative distance is less than the preset distance threshold, it means that the target customer has a purchase intention, and at this time, the customer dynamic video including the target customer is obtained by an image acquisition device disposed outside the intelligent unmanned vending cabinet. The commonly used image acquisition device is a high-definition camera.

[0087] S103: Analyze the customer's dynamic video for abnormal behavior, and determine whether the target customer has a first abnormal behavior based on the abnormal behavior analysis result.

[0088] In this embodiment, the first abnormal behavior refers to some customers' violent behaviors such as shaking and slapping the intelligent unmanned vending machine. By analyzing the customer dynamic video frame by frame, the local behavior features and global behavior features of the target customer in each frame of the customer behavior image are obtained. After normalizing the local behavior features and the global behavior features, the local feature descriptor of each interest point neighborhood in the local behavior features is extracted, and the frame of customer behavior image corresponding to each interest point neighborhood is found. The optical flow histogram of the corresponding frame of customer behavior image is spliced ​​with the local feature descriptor to obtain a larger feature vector. The same method is used to traverse the interest point neighborhoods of all interest points, and the local feature descriptors are spliced ​​with the corresponding light flow histograms. After completion, all the obtained feature vectors are integrated to obtain the character behavior features of the target customer. In addition to completing feature fusion by splicing feature vectors, feature fusion can also be performed using weighted fusion, convolution fusion and other methods to obtain character behavior features that describe the complete behavior features of the target customer. After obtaining the target customer's character behavior characteristics, the abnormal behavior recognition is performed using the pre-built abnormal behavior recognition model, and the probability of belonging to the first abnormal behavior, that is, the abnormal output value, is output. If the abnormal output value is greater than the preset output threshold (which can be 0.7), it means that the target customer has the first abnormal behavior. In addition, the pre-built abnormal behavior recognition model is constructed based on a neural network model. Commonly used neural network models include long short-term memory network models, recurrent neural network models, etc.

[0089] S104. If the target customer has the first abnormal behavior, the smart unmanned vending machine is adjusted from the normal operating state to the cabinet door locked state within a preset time, and the cabinet door cannot be unlocked in the cabinet door locked state.

[0090] In this embodiment, if the recognition result of the abnormal behavior recognition model shows that the target customer has a first abnormal behavior such as kicking the cabinet door or slapping the cabinet door, the smart unmanned vending cabinet will immediately lock the cabinet door and adjust itself from a normal state to a cabinet door locked state. The cabinet door locked state will be maintained for a period of time. During this period of time, the target customer cannot unlock the cabinet door by scanning a code or other means. At the same time, when the first abnormal behavior is detected, the alarm system inside the smart unmanned vending cabinet will transmit the dynamic video of the customer with the first abnormal behavior and the location information of the smart unmanned vending cabinet to the management platform of the smart unmanned vending cabinet, so that the staff of the management platform can maintain the smart unmanned vending cabinet in time and extend the service life of the smart unmanned vending cabinet.

[0091] S105. If the target customer does not have the first abnormal behavior, the normal operation of the intelligent unmanned vending machine is maintained.

[0092] In this embodiment, if the recognition result of the abnormal behavior recognition model shows that the target customer does not have the first abnormal behavior, the smart unmanned vending machine does not need to lock the door, and maintains a normal operating state that can be unlocked by the target customer.

[0093] S106. Obtain sensor information of the smart unmanned vending cabinet during the information collection period through sensors installed inside the smart unmanned vending cabinet. The sensor information includes gravity sensor information and image sensor information. The starting point of the information collection period is the initial time node when the target customer completes unlocking the cabinet door, and the ending point of the information collection period is the deadline time node when the smart unmanned vending cabinet completes closing the cabinet door.

[0094] In this embodiment, the sensors arranged inside the intelligent unmanned vending cabinet include image sensors and gravity sensors, wherein the image sensors include high-definition cameras, and each layer inside the intelligent unmanned vending cabinet is provided with high-definition cameras, and gravity sensors are arranged on the shelves on each layer. When customers purchase goods, the images taken by the image sensors and the gravity changes detected by the gravity sensors can accurately identify the types and quantities of goods purchased by the customers. The gravity sensor information is the gravity change data detected by the gravity sensor. When customers take and put down goods, the gravity sensor can detect the gravity changes of the shelves. The image sensor information is the image of the goods on each layer, and the type of items taken by the customers can be detected by the image sensor. In addition, there is a communication connection relationship between the gravity sensors and image sensors on each layer inside the intelligent unmanned vending cabinet, and the gravity sensor information and image sensor information refer to the information collected by all gravity sensors and image sensors. The information collection time period is also the time period when the target customers select goods. Only the sensor information in the time period from the target customers completing the door unlocking to the door closing is obtained to reduce unnecessary computing power waste and improve the efficiency of the subsequent second abnormal behavior analysis. In addition, the cabinet door and body of the smart unmanned vending machine are equipped with magnets and magnetic sensors respectively, which can detect in real time whether the cabinet door is closed.

[0095] S107. Identify a second abnormal behavior of the target customer within the information collection time period by combining the gravity sensor information and the image sensor information.

[0096] In this embodiment, the gravity sensor information is first used to analyze the gravity changes during the time period when the customer is shopping for goods, and a preliminary judgment is made based on the gravity changes whether the target customer has a second abnormal behavior. If the gravity changes fluctuate, it means that the target customer has a complex behavior of picking up and putting down goods when shopping for goods. At this time, it is necessary to further combine the image sensor information to judge the second abnormal behavior. The second abnormal behavior includes the target customer putting items that do not belong to the smart unmanned vending cabinet into the smart unmanned vending cabinet, and the target customer replacing the goods in the smart unmanned vending cabinet with items that do not belong to the smart unmanned vending cabinet.

[0097] When it is identified that the target customer has a second abnormal behavior, the customer dynamic video of the target customer during the information collection time period is immediately obtained, and the alarm system set up inside the smart unmanned vending cabinet will transmit the customer dynamic video containing the second abnormal behavior and the location information of the smart unmanned vending cabinet to the management platform of the smart unmanned vending cabinet, so that the platform staff can check and replace the goods in time.

[0098] In one embodiment, performing abnormal behavior analysis on a customer dynamic video and determining whether a target customer has a first abnormal behavior according to the abnormal behavior analysis result includes the following steps:

[0099] The customer dynamic video is input into a pre-built visual saliency model, and the customer dynamic video is analyzed for saliency through the visual saliency model to obtain the behavior saliency values ​​of all behavior pixels in each frame of the customer behavior image in the customer dynamic video;

[0100] For any frame of customer behavior image, the behavior pixel points whose behavior significance value in the customer behavior image is greater than a preset significance value threshold are screened out to obtain a plurality of significant pixel points;

[0101] Integrate all the significant pixels to obtain the character area of ​​the customer behavior image;

[0102] A dynamic behavior filter is used to perform convolution analysis on the customer behavior image in the area with people, to obtain a behavior response map of the customer behavior image, and based on the behavior response map, the person interest points of the customer behavior image are calculated;

[0103] The local feature extraction algorithm is used to calculate the local gradient of all interest point neighborhoods of the customer behavior image. The interest point neighborhood is the image area within a preset range around the person's interest point.

[0104] Calculate the local feature descriptors of all interest point neighborhoods based on local gradients, and integrate all local feature descriptors to obtain the local behavior features of each customer behavior image;

[0105] The optical flow estimation algorithm is used to calculate the optical flow histogram of all customer behavior images, and all optical flow histograms are integrated to obtain the global behavior characteristics of the target customers;

[0106] The local behavior features corresponding to the customer behavior image are fused with the corresponding global behavior features to obtain the character behavior features of the target customer;

[0107] The character behavior characteristics are input into a pre-built abnormal behavior recognition model to perform abnormal behavior recognition, obtain an abnormal output value, and determine whether the target customer has the first abnormal behavior based on the abnormal output value.

[0108] In this embodiment, the visual saliency model is a model that simulates human visual characteristics through an intelligent algorithm to extract salient areas in an image. The main steps of calculating the behavior saliency value using the visual saliency model are to first calculate the color value and brightness value of each behavior pixel in each frame of the customer behavior image, calculate the similarity between the behavior pixels using the Euclidean distance formula and the color value and brightness value of each behavior pixel, take the behavior pixel as the node of the graph, and use the similarity between the behavior pixels as the weight of the edge, select some nodes as initial nodes, and assign initial behavior saliency values ​​to the initial nodes according to the similarity, perform random walks between each node using the random walk model, calculate the walk probability based on the weight of the edge between nodes, and update and iterate the behavior saliency value of each node according to the walk probability until the preset maximum number of iterations is reached, and obtain the behavior saliency values ​​of all behavior pixels in the customer behavior image. The random walk model is a classic time series model, which is usually used to describe the random changes of a variable over time. Its core concept is that the conservation quantity carried by any irregular walker corresponds to a diffusion transport law.

[0109] The behavior pixels in each frame of the customer behavior image whose behavior significance value is greater than the preset significance value threshold are screened out, and the significant area in the customer behavior image, that is, the person area, can be obtained based on the screened significant pixels. The person area obtained at this time is the approximate area of ​​the target customer, so it is necessary to further use a dynamic behavior filter to screen out the person interest points in the customer behavior image.

[0110] The dynamic behavior filter is a comprehensive filter composed of two different types of filters, which can be composed of a Gaussian filter and a Gabor filter. First, the customer behavior image with the human figure area is input into the Gaussian filter, and the image is smoothed by convolution operation. Then, the customer behavior image is convolved with two Gabor filters in different directions to capture the features of the customer behavior image in the vertical and horizontal directions, and two response maps in different directions are obtained. After combining them, the behavior response map of the customer behavior image is obtained. The behavior response map reflects the feature intensity of the customer behavior image in a specific direction and frequency, which is an important basis for subsequent feature analysis and interest point detection. Then, the behavior response values ​​of all behavior pixels in the behavior response map are calculated according to the response function, and different pixel thresholds are assigned to the human figure area and non-human figure area in the customer behavior image according to the pixel threshold formula. Compared with the uniform pixel threshold, different pixel thresholds corresponding to different areas can more accurately screen out the human interest points, which is convenient for the subsequent accurate identification of whether the target customer has the first abnormal behavior.

[0111] The local gradients of all points of interest in the customer behavior image are calculated using a local feature extraction algorithm (such as a scale-invariant feature transformation algorithm), and then the neighborhood is divided for each point of interest according to a preset range to obtain the neighborhood of the point of interest. The local gradient of the neighborhood of the point of interest is obtained according to the local gradient of each point of interest. The gradient direction histogram of the neighborhood of each point of interest is calculated according to the local gradient of the neighborhood of the point of interest. The gradient direction histogram is the local feature descriptor. The local feature descriptors in each frame of the customer behavior image are integrated to obtain the local behavior features of the customer behavior image. Then, the optical flow estimation algorithm is used to calculate the optical flow histogram of the customer behavior image. The optical flow histogram can represent the motion information of the target customer, make up for the locality of the local features, and facilitate the subsequent behavior recognition of the target customer.

[0112] After obtaining the local behavior features and the global behavior features, the feature fusion algorithm is used to fuse the local behavior features and the global behavior features. Commonly used feature fusion algorithms include splicing method, weighted fusion method and multi-scale feature fusion algorithm. Taking the splicing method as an example, the steps of feature fusion are as follows:

[0113] 1.1. Normalize the local behavior features and the global behavior features. You can use standard deviation normalization or deviation normalization to ensure that the local behavior features and the global behavior are on the same scale;

[0114] 1.2. Extract the local feature descriptor of each interest point neighborhood in the local behavior feature, and find the optical flow histogram of the customer behavior image where each interest point neighborhood is located;

[0115] 1.3. Concatenate each optical flow histogram with the feature vector of the corresponding local feature descriptor one by one to obtain a new feature vector. For example, the feature vector of the local feature descriptor is (X1, X2, ..., X N ), the feature vector of the optical flow histogram is (Y1,Y2,……,Y M ), where N is the dimension of the local feature descriptor, M is the bin value (number of intervals) of the optical flow histogram, and the new feature vector obtained after splicing is (X1, X2, …, X N ,Y1,Y2,……,Y M ).

[0116] After all the optical flow histograms and local feature descriptors are spliced, feature fusion is completed and the character behavior characteristics of the target customers are obtained.

[0117] The obtained character behavior characteristics are input into the pre-built abnormal behavior recognition model for abnormal behavior recognition, and the probability of belonging to the first abnormal behavior, that is, the abnormal output value, is output. If the abnormal output value is greater than the preset output threshold (which can be 0.7), it means that the target customer has the first abnormal behavior. In addition, the pre-built abnormal behavior recognition model is built based on a neural network model. Commonly used neural network models include long short-term memory network models, recurrent neural network models, etc. Taking the long short-term memory network model as an example, the steps for building and training an abnormal behavior recognition model based on the long short-term memory network model are as follows:

[0118] 2.1. Obtain multiple historical customer dynamic videos containing the first abnormal behavior from the historical database, annotate the normal behavior and the first abnormal behavior in the historical customer dynamic videos, and randomly divide them into an abnormal training set and an abnormal verification set;

[0119] 2.2. Perform feature extraction and feature fusion on each historical customer dynamic video in the abnormal training set to obtain a feature sequence;

[0120] 2.3. Define the layer structure of the abnormal behavior recognition model, including setting the number of LSTM layers (3 layers can be set), selecting a suitable activation function (logic function can be used), etc.

[0121] 2.4. Prepare for model training, select a suitable loss function, such as cross entropy loss, to measure model performance, select a suitable optimization algorithm, such as stochastic gradient descent algorithm, adaptive algorithm, etc., and set parameters such as the maximum number of training times and learning rate;

[0122] 2.5. Input the feature sequence of the abnormal training set into the abnormal behavior recognition model for forward propagation, and iteratively optimize the parameters until the number of iterative optimizations reaches the maximum number of training times;

[0123] 2.6. After reaching the maximum number of training times, the abnormal behavior recognition model is verified using the abnormal verification set to evaluate the model's accuracy, recall rate and other indicators. When the various indicators of the abnormal behavior recognition model reach the preset indicator threshold, it means that the abnormal behavior recognition model is trained.

[0124] In one embodiment, using a dynamic behavior filter to perform convolution analysis on a customer behavior image in which a person region exists to obtain a behavior response graph of the customer behavior image, and calculating the person interest points of the customer behavior image based on the behavior response graph includes the following steps:

[0125] Input the customer dynamic video with the character area marked into the dynamic behavior filter for filtering;

[0126] Perform convolution processing in different directions on each frame of the customer behavior image of the customer dynamic video that has completed the filtering processing, and obtain a horizontal response map and a vertical response map of each frame of the customer behavior image;

[0127] Combine the horizontal response graph and the vertical response graph corresponding to all customer behavior images to obtain the behavior response graph of all customer behavior images;

[0128] Calculate the behavior response values ​​of all behavior pixels in the behavior response graph using the response function;

[0129] For each frame of customer behavior image, the sum of the behavior saliency values ​​of all behavior pixels in the person area and non-person area of ​​the customer behavior image is calculated to obtain the regional saliency value and the non-regional saliency value, wherein the non-person area is the area other than the person area in the customer behavior image;

[0130] According to the total regional saliency values ​​and non-regional saliency values, weights are assigned to the character region and the non-character region respectively;

[0131] The pixel threshold formula is used to assign thresholds to the behavior pixels in the character area and non-character area that have completed the weight assignment, and the regional response threshold and non-regional response threshold are obtained. The pixel threshold formula is as follows:

[0132]

[0133]

[0134] Among them, D a is the regional response threshold, D b is the non-regional response threshold, is the regional saliency value of the customer behavior image corresponding to the i-th time node, is the non-regional saliency value of the customer behavior image corresponding to the i-th time node, ∈ is a preset fixed value, γ is a preset time period, β α and β b are the weights of the character area and the non-character area, t is the time coordinate of the customer behavior image corresponding to the behavior response graph, G a is the sum of all regional saliency values ​​of all customer behavior images, G b It is the sum of all non-regional saliency values ​​of all customer behavior images;

[0135] For each character region in all customer behavior images, the behavior pixel points in the character region whose behavior response value is greater than the region response threshold are screened out as the character interest points of the customer behavior image;

[0136] The behavior pixel points whose behavior response values ​​in the non-person area are greater than the non-region response threshold are screened out as the person interest points of the customer behavior image.

[0137] In this embodiment, the dynamic behavior filter is a comprehensive filter composed of two different types of filters, which can be composed of a Gaussian filter and a Gabor filter. First, the customer behavior image with the character area is input into the Gaussian filter, and the noise in the image is reduced and the image is smoothed by convolution operation. Specifically, for each behavior pixel in the customer behavior image, the weight of the Gaussian filter is used to perform weighted summation with the surrounding pixels. Then, the customer behavior image is convolved with two Gabor filters in different directions to capture the features of the customer behavior image in the vertical and horizontal directions, and two response graphs in different directions are obtained, which are combined to obtain the behavior response graph of the customer behavior image. Among them, the Gabor filter used is composed of a sine wave and a Gaussian envelope function. The two Gabor filters in different directions are perpendicular to each other in direction, for example, one is 0° and the other is 90°. One of the filters (such as 0°) is used to convolve the customer behavior image to obtain a horizontal response graph, and the other filter (such as 90°) is used to convolve the customer behavior image to obtain a vertical response graph. The horizontal response graph and the horizontal and vertical response graph are weighted averaged to obtain the behavior response graph of the customer behavior image. The behavior response map can highlight specific features in the customer behavior image, such as edges, textures or other structures, and reflects the feature strength of the customer behavior image in a specific direction and frequency. It is an important basis for subsequent feature analysis and interest point detection. Then, the behavior response values ​​of all behavior pixels in the behavior response map are calculated according to the response function, and different pixel thresholds are assigned to the human area and non-human area in the customer behavior image according to the pixel threshold formula. Compared with the unified pixel threshold, assigning different pixel thresholds to different areas can more accurately screen out human interest points, which is convenient for subsequent accurate identification of whether the target customer has the first abnormal behavior.

[0138] The response function is used to calculate the behavior response values ​​of all behavior pixels in the behavior response graph. The response function is as follows:

[0139]

[0140] Among them, H(x, y) is a two-dimensional Gaussian smoothing kernel, (x, y) is the pixel coordinate of the behavior pixel on the behavior response map, σ is the preset parameter of the Gaussian filter, ω and τ are the preset parameters of the Gabor filter, and T is the behavior response map.

[0141] After obtaining the behavior response values ​​of all behavior pixels, the sum of the behavior saliency values ​​of all the behavior pixels in the person area and the non-person area is calculated respectively to obtain the regional saliency value and the non-regional saliency value, wherein the non-person area is the area other than the person area in the customer behavior image. The main function of this step is to facilitate the subsequent calculation of the pixel threshold. At the same time, weights can be assigned to the person area and the non-person area respectively according to the regional saliency value and the non-regional saliency value, so that the calculated pixel threshold is more reliable.

[0142] After completing the weight assignment, the pixel threshold formula is used to assign thresholds to the behavior pixels in the human area and non-human area that have completed the weight assignment, and the regional response threshold and non-regional response threshold are obtained. The regional response threshold is the threshold used to filter the human interest points in the human area, and the non-regional response threshold is the threshold used to filter the human interest points in the non-human area.

[0143] When the interest points are screened for the person area in the customer behavior image, the behavior pixel points in the person area whose behavior response value is greater than the regional response threshold are screened out using the regional response threshold to obtain the person interest points of the customer behavior image. Similarly, the behavior pixel points in the non-person area whose behavior response value is greater than the non-regional response threshold are screened out to obtain the person interest points.

[0144] In one embodiment, calculating the optical flow histogram of all customer behavior images using an optical flow estimation algorithm, and integrating all the optical flow histograms to obtain the global behavior characteristics of the target customer includes the following steps:

[0145] Extracting multiple frames of continuous customer behavior images from the customer dynamic video in time sequence as candidate behavior images;

[0146] Calculate the optical flow vectors of the interest points of the characters in all candidate behavior images using the optical flow estimation algorithm;

[0147] According to the optical flow vectors of the character's interest points, the optical flow amplitude and optical flow direction of all the character's interest points are calculated;

[0148] Construct an optical flow histogram for all candidate behavior images according to the optical flow amplitude and optical flow direction;

[0149] After traversing all customer behavior images in the customer dynamic video, all optical flow histograms are integrated to obtain the global behavior characteristics of the target customer.

[0150] In this embodiment, optical flow is the instantaneous speed of the pixel movement of a spatial moving object on the observation imaging plane. It is a method of finding the corresponding relationship between the previous frame and the current frame by using the change of pixels in the image sequence in the time domain and the correlation between adjacent frames, thereby calculating the motion information of the object between adjacent frames. Common optical flow estimation algorithms include the Lucas-Kanade method (optical flow estimation method based on local areas), the Horn-Schunck method (optical flow estimation method based on global areas), etc.

[0151] Specifically, multiple frames (can be two frames) of continuous customer behavior images are extracted from the customer dynamic video in chronological order as candidate behavior images. This step is to obtain a short section of continuous customer behavior images to facilitate the analysis of customer motion characteristics in customer behavior images, and then use the optical flow estimation algorithm to calculate the optical flow vectors of the character interest points in all candidate behavior images. Taking the Lucas-Kanade method as an example, the Lucas-Kanade method is based on two assumptions, namely the brightness invariance assumption and the neighborhood optical flow similarity assumption. The Lucas-Kanade method uses these equations to solve the optical flow by assuming that all pixels in a 3x3 field have the same motion. After obtaining the optical flow vector, the optical flow amplitude and optical flow direction of the character interest point are calculated according to the optical flow vector. For example, if the optical flow vector is (a, b), the optical flow amplitude is The direction of the optical flow is arctan(b / a). The optical flow direction is divided into multiple intervals, for example, 0° to 360° is divided into 8 or 16 buckets, and the sum of the optical flow amplitudes in each bucket is calculated. The sum of the amplitudes of each bucket is filled into the optical flow histogram to form an optical flow histogram that describes the global motion characteristics of the person. The optical flow histogram of all customer behavior images is calculated, and all optical flow histograms are integrated to obtain the global behavior characteristics of the target customer.

[0152] In one embodiment, combining the gravity sensor information and the image sensor information to identify the second abnormal behavior of the target customer during the information collection time period includes the following steps:

[0153] Extract the gravity change characteristics of the gravity sensor information at the cut-off time node, and analyze the gravity change trend of the intelligent unmanned vending machine based on the gravity change characteristics;

[0154] If the gravity change trend is that the gravity continues to increase, it is judged that the target customer has a second abnormal behavior during the information collection period;

[0155] If the gravity change trend is that the gravity continues to decrease, it is determined that the target customer does not have a second abnormal behavior during the information collection period;

[0156] If the gravity change trend is that the gravity fluctuates up and down, the gravity difference between the initial time node and the end time node is calculated, and the gravity difference and the image sensor information are combined to identify the second abnormal behavior of the target customer during the information collection time period.

[0157] In this embodiment, refer to Figure 2 At the moment when the door of the smart unmanned vending cabinet is closed, that is, the deadline, the gravity change feature of the gravity sensor information within the information collection period is immediately extracted, and the gravity change trend of the smart unmanned vending cabinet is analyzed based on the gravity change feature. This is because when the target customer takes or puts down the goods, the gravity sensor installed on the shelf can detect the gravity change of this shelf, so it can be inferred whether the customer takes or puts down the goods.

[0158] If the gravity change trend shows that the gravity continues to increase during the information collection time period, it means that the target customer has placed something in the smart unmanned vending machine. This behavior is not allowed and will affect the shopping experience of subsequent customers. This behavior of placing other items in the smart unmanned vending machine is a second abnormal behavior. Therefore, it is judged that the target customer has the second abnormal behavior during the information collection time period.

[0159] If the gravity change trend shows that the gravity continues to decrease during the information collection time period, it means that the target customer is purchasing goods normally. Therefore, it is judged that the target customer does not have a second abnormal behavior during the information collection time period. Subsequently, the type and quantity of goods purchased by the target customer are analyzed based on the sensor information, and the bill information is sent to the target customer.

[0160] If the gravity change trend shows that the gravity fluctuates up and down during the information collection time period, it means that the target customer has taken and put down goods or other items in the process of purchasing goods. If the target customer simply takes the goods and then puts the goods back to the smart unmanned vending machine for various reasons, this is a normal behavior. However, some customers may replace or damage the goods in the smart unmanned vending machine, such as taking a stone to replace the drink in the smart unmanned vending machine or taking out the drink and drinking part of it and putting it back. These are all second abnormal behaviors, but they cannot be judged only by the gravity change trend. Therefore, it is necessary to calculate the gravity difference between the initial time node and the end time node, and combine the gravity difference and image sensor information to further identify whether the target customer has second abnormal behavior in the process of purchasing goods.

[0161] In one embodiment, calculating the gravity difference between the initial time node and the end time node, and identifying the second abnormal behavior of the target customer during the information collection time period by combining the gravity difference and the image sensor information includes the following steps:

[0162] Obtain the initial gravity of the initial time node and the cutoff gravity of the cutoff time node according to the gravity sensor information;

[0163] Determine whether the gravity difference obtained by subtracting the initial gravity from the cut-off gravity is zero. If the gravity difference is zero, identify the second abnormal behavior of the target customer during the information collection period through the image sensor information;

[0164] If the gravity difference is not zero, then analyzing whether the gravity difference is greater than zero; if the gravity difference is less than zero, then identifying the second abnormal behavior of the target customer during the information collection period through the image sensor information;

[0165] If the gravity difference is greater than zero, it is determined that the target customer has a second abnormal behavior during the information collection time period.

[0166] In this embodiment, if Figure 2 As shown, the initial gravity of the initial time node is extracted from the gravity sensor information, when the customer has not started to purchase the goods, and the cut-off gravity of the cut-off time node is extracted, when the customer has completed the purchase of the goods. The cut-off gravity is subtracted from the initial gravity to obtain the gravity difference. If the gravity difference is zero, it means that the target customer took the goods in the process of purchasing goods, but eventually put all the goods back into the smart unmanned vending cabinet for various reasons. This is a normal behavior, but there is a possibility that the target customer may have taken the goods, but the weight of the items put back into the smart unmanned vending cabinet is exactly the same as the weight of the goods taken. Therefore, it is necessary to further perform image recognition on the image sensor information. By comparing the changes in the product image before the target customer purchases the goods and the product image after the goods are purchased, it is possible to identify whether there are other items in the smart unmanned vending cabinet, that is, items that do not belong to the smart unmanned vending cabinet.

[0167] If the gravity difference is not zero, further analysis is performed to determine whether the gravity difference is greater than zero or less than zero. If the gravity difference is less than zero, it means that the target customer has taken the goods but put some of the goods or other items into the smart unmanned vending machine. Therefore, image recognition of the image sensor information is still required to identify whether there are items in the smart unmanned vending machine that do not belong to the smart unmanned vending machine.

[0168] If the gravity difference is greater than zero, it means that after taking the goods, the target customer placed other items in the smart unmanned vending machine that are heavier than the goods. As a result, although the target customer selected the goods, the weight of the items on the shelf increased. It can be directly judged that the target customer used other items to replace the goods in the smart unmanned vending machine, which is the second abnormal behavior. Therefore, it can be directly judged that the target customer has the second abnormal behavior during the information collection time period.

[0169] In one embodiment, identifying the second abnormal behavior of the target customer during the information collection time period through image sensor information includes the following steps:

[0170] The image sensor information collected at the initial time node is used as the reference image information, and the image sensor information collected at the cutoff time node is used as the target image information;

[0171] Using image registration technology to register the reference image information and the target image information;

[0172] Extracting the color histogram of the reference image information and the target image information after image registration is completed;

[0173] Grayscale processing is performed on the reference image information and target image information that have completed the registration;

[0174] The edge detection algorithm is used to extract the contours of the product in the reference image information and the target image information after grayscale processing respectively;

[0175] Acquire shape features of reference image information and target image information according to the product outline;

[0176] Comparing the color histogram and shape features between the reference image information and the target image information respectively, if the color histogram and shape features are the same, it is determined that the target customer does not have a second abnormal behavior during the information collection period;

[0177] If the color histogram and / or shape features are different, it is determined that the target customer has a second abnormal behavior during the information collection time period.

[0178] In this embodiment, if Figure 2As shown, the image sensor information collected at the initial time node is used as the reference image information. At the initial time node, the target customer has not yet purchased the product, so the image sensor information collected at this time is used as the reference image, and the image sensor information collected at the deadline time node is used as the target image information. At this time, the target customer has completed the purchase of the product, so the image sensor information at the deadline time node is used as the image to be detected. First, the reference image information and the target image information are registered using image registration technology to ensure that the reference image information and the target image information are as consistent as possible in perspective and position, so as to facilitate subsequent comparison. Commonly used image registration technologies include feature matching, affine transformation, etc. Taking affine transformation as an example, the two images are aligned by translating, rotating, scaling and shearing the target image information. After completing the image registration, the reference image information and the target image information are converted from RGB to HSV color space respectively, and the HSV color space of the two images is divided into multiple fixed intervals according to the color range. Then the histogram of each color area is calculated to obtain the color histograms of the reference image information and the target image information. The color histograms of the two images can reflect the color differences of the reference image information and the target image information.

[0179] Then, the aligned reference image information and target image information are grayed out. This step is to highlight the contrast between the two and facilitate the subsequent edge extraction of the two. After both are grayed out, the edge points of the reference image information and the target image information are detected respectively using the edge detection algorithm. All edge points are clustered and integrated, and the edge segmentation is performed to obtain the product contours of all products. Commonly used edge detection algorithms include the Sobel operator, the differential operator or the Roberts operator. Taking the Sobel operator as an example, the Sobel operator detects the edge by calculating the gradient of the image brightness. Two 3x3 convolution kernels are used to detect horizontal edges and vertical edges respectively, thereby obtaining the edge points of the image. After obtaining the product contours of the reference image information and the target image information, the shape features of the products are extracted according to the product contours of the two. The shape features include perimeter, area, shape moment, etc.

[0180] After obtaining the color histogram and shape features of the reference image information and the target image information, the color histogram and shape features of the two are compared to see if they are the same. The color histograms of the reference image information and the target image information can be compared to see if they are the same using methods such as cosine similarity or histogram intersection. Taking the histogram intersection method as an example, the similarity of the two color histograms is measured by calculating the degree of overlap of the two color histograms in each color interval. The calculation formula for the degree of overlap is as follows:

[0181]

[0182] Among them, H1 is the color histogram of the reference image information, H2 is the color histogram of the target image information, min(H1[i], H2[i]) means taking the minimum value of the i-th bin (color interval) value of the two color histograms, and n refers to the number of color intervals.

[0183] The shape features of the reference image information and the target image information can be compared using methods such as Hausdorff distance and shape context to compare the similarity of the shape features of the two. Taking the Hausdorff distance method as an example, the edge points extracted in the previous article are used as boundary point sets to obtain the boundary point sets of the reference image information and the boundary point sets of the target image information. For each point in each point set, the distance to all points in the other point set is calculated, and the distance from each point to the nearest point of the other point set is found. The distance of the nearest point obtained is the Hausdorff distance. The smaller the Hausdorff distance, the higher the similarity of the shape features of the two.

[0184] If the calculated overlap degree is greater than the preset degree threshold and the Hausdorff distance is less than the preset minimum distance threshold, it means that the target customer has not replaced or damaged the goods in the smart unmanned vending cabinet, and it is determined that the target customer has no second abnormal behavior in the process of purchasing goods. If the calculated overlap degree is less than the preset degree threshold or the Hausdorff distance is greater than the preset minimum distance threshold, it means that other items that do not belong to the smart unmanned vending cabinet appear in the smart unmanned vending cabinet, so it is determined that the target customer has a second abnormal behavior.

[0185] In one embodiment, the method further comprises:

[0186] If the target customer does not have the second abnormal behavior, the reference image information and the target image information are respectively input into a preset commodity recognition model to obtain reference commodity information and target commodity information;

[0187] Compare the benchmark product information with the target product information to obtain the types and quantities of products sold by the smart unmanned vending machine;

[0188] Calculate the weight of goods sold based on the types and quantities of goods sold;

[0189] Compare the absolute value of the difference between the weight of the sold goods and the gravity difference to see if they are the same. If the weight of the sold goods and the absolute value of the difference are not the same, input the reference product information and the target product information into the product recognition model again until the weight of the sold goods and the absolute value of the difference are the same.

[0190] If the weight of the goods sold is the same as the absolute value of the difference, the target customer's billing information is output based on the type and quantity of goods sold;

[0191] If the target customer has a second abnormal behavior, the intelligent unmanned vending machine is controlled to output an alarm message.

[0192] In this embodiment, if the target customer does not have the second abnormal behavior, the types and quantities of goods purchased by the target customer are analyzed through sensor information, the price payable by the customer is calculated based on the types and quantities of goods, and the bill information containing the types, quantities, prices and total prices of goods is sent to the target customer. Specifically, the reference image information and the target image information are first input into a preset commodity recognition model. The commodity recognition model is constructed based on a neural network model (such as a convolutional neural network model). Taking the convolutional neural network model as an example, a variety of commodity images with completed commodity category annotations are used as training sets, and the training set contains various commodity categories and changes (such as lighting, angles, etc.). The training set is processed by image enhancement and the like and then input into the commodity recognition model for model training. The commodity recognition model includes a convolutional layer, an activation layer, a pooling layer, and a fully connected layer, wherein the convolutional layer performs feature extraction, the activation layer introduces nonlinear factors, the pooling layer reduces the dimension of the commodity images in the training set, and the fully connected layer maps the features to specific commodity categories. After the commodity recognition model is updated and iterated multiple times, a trained commodity recognition model is obtained. After the reference image information and the target image information are respectively input into the trained commodity recognition model, the item categories contained in the two are output respectively, and the number of occurrences of the same item category is counted to obtain the reference commodity information and the target commodity information respectively. The commodity types and commodity quantities contained in the target commodity information are compared with the commodity types and commodity quantities in the benchmark commodity information. The commodity types and commodity quantities missing from the target commodity information are the commodity types and commodity quantities sold by the smart unmanned vending machine.

[0193] Since some products may be inaccurately identified due to occlusion during the identification process, it is necessary to compare whether the absolute value of the difference between the weight of the sold product and the gravity difference is the same. The weight of the product can be directly calculated based on the product weight information stored in the database and the corresponding number of products. If the weight of the sold product is not the same as the absolute value of the difference, the reference image information and the target image information are input into the product recognition model again to identify the type of product until the weight of the sold product is the same as the absolute value of the difference. If the weight of the sold product is the same as the absolute value of the difference, it means that the identification result of the product type is accurate. Therefore, the target customer's bill information can be directly calculated based on the acquired type of sold product, the number of sold products, and the pre-stored product prices corresponding to different types of products, and the bill information is sent to the target customer.

[0194] If the target customer is identified to have the second abnormal behavior, the dynamic video of the target customer during the information collection period is immediately obtained, and the alarm system inside the smart unmanned vending cabinet is set to output an alarm message, which will be transmitted to the management platform of the smart unmanned vending cabinet. The alarm message includes the dynamic video of the customer with the second abnormal behavior and the location information of the smart unmanned vending cabinet. When the management platform receives the alarm message, it will dispatch the management personnel of the smart unmanned vending cabinet to inspect the smart unmanned vending cabinet, and replace the damaged or replaced abnormal products to ensure the shopping experience of other customers.

[0195] The present application also discloses an interactive behavior recognition system for an intelligent unmanned vending machine, which is characterized by comprising:

[0196] a memory configured to store instructions; and

[0197] The processor is configured to call instructions from the memory and implement any of the above-mentioned methods for interactive behavior recognition of intelligent unmanned vending machines when executing the instructions.

[0198] Among them, the processor can adopt a central processing unit (CPU). Of course, according to actual usage, other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. can also be adopted. The general-purpose processor can adopt a microprocessor or any conventional processor, etc., and this application does not impose any restrictions on this.

[0199] Among them, the memory can be an internal storage unit of a computer device, such as a hard disk or memory of a computer device, or an external storage device of a computer device, such as a plug-in hard disk, a smart memory card (SMC), a secure digital card (SD) or a flash memory card (FC) equipped on the computer device, etc., and the memory can also be a combination of an internal storage unit and an external storage device of a computer device. The memory is used to store computer programs and other programs and data required by the computer device. The memory can also be used to temporarily store data that has been output or is to be output, and this application does not impose any restrictions on this.

[0200] An embodiment of the present application also provides a machine-readable storage medium, on which instructions are stored, and the instructions are used to enable a machine to execute the above-mentioned method for interactive behavior recognition for a smart unmanned vending machine.

[0201] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0202] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0203] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0204] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0205] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0206] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0207] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0208] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0209] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A method for identifying interactive behaviors of an intelligent unmanned vending machine, characterized in that: The method comprises the following steps: Detecting the relative distance between the target customer and the smart unmanned vending cabinet by an infrared sensor disposed outside the smart unmanned vending cabinet; When the relative distance is less than a preset distance threshold, a customer dynamic video including the target customer is acquired by an image acquisition device disposed outside the intelligent unmanned vending cabinet; Performing abnormal behavior analysis on the customer dynamic video, and determining whether the target customer has a first abnormal behavior according to the abnormal behavior analysis result; If the target customer has the first abnormal behavior, the smart unmanned vending machine is adjusted from a normal operating state to a cabinet door locked state within a preset time, and the cabinet door cannot be unlocked in the cabinet door locked state; If the target customer does not have the first abnormal behavior, maintaining the normal operating state of the smart unmanned vending machine; The sensor information of the smart unmanned vending cabinet within the information collection time period is obtained by a sensor disposed inside the smart unmanned vending cabinet, wherein the sensor information includes gravity sensor information and image sensor information, the starting point of the information collection time period is the initial time node when the target customer completes unlocking the cabinet door, and the ending point of the information collection time period is the deadline time node when the smart unmanned vending cabinet completes closing the cabinet door; The gravity sensor information and the image sensor information are combined to identify a second abnormal behavior of the target customer within the information collection time period.

2. The method according to claim 1, characterized in that The abnormal behavior analysis of the customer dynamic video and judging whether the target customer has a first abnormal behavior according to the abnormal behavior analysis result comprises the following steps: Inputting the customer dynamic video into a pre-built visual saliency model, performing saliency analysis on the customer dynamic video through the visual saliency model, and obtaining behavior saliency values ​​of all behavior pixels in each frame of the customer behavior image in the customer dynamic video; For any frame of the customer behavior image, the behavior pixel points whose behavior significance value in the customer behavior image is greater than a preset significance value threshold are screened out to obtain a plurality of significant pixel points; Integrate all the significant pixel points to obtain the character area of ​​the customer behavior image; Using a dynamic behavior filter to perform convolution analysis on the customer behavior image where the character area exists, to obtain a behavior response map of the customer behavior image, and to calculate the character interest points of the customer behavior image based on the behavior response map; Calculating the local gradient of all interest point neighborhoods of the customer behavior image using a local feature extraction algorithm, wherein the interest point neighborhood is an image area within a preset range around the person's interest point; Calculate the local feature descriptors of all the neighborhoods of the interest points according to the local gradients, and integrate all the local feature descriptors to obtain the local behavior features of each of the customer behavior images; Calculating the optical flow histograms of all the customer behavior images using an optical flow estimation algorithm, and integrating all the optical flow histograms to obtain the global behavior characteristics of the target customer; Fusing the local behavior feature corresponding to the customer behavior image with the corresponding global behavior feature to obtain the character behavior feature of the target customer; The character behavior feature is input into a pre-built abnormal behavior recognition model to perform abnormal behavior recognition to obtain an abnormal output value, and it is determined whether the target customer has a first abnormal behavior according to the abnormal output value.

3. The method according to claim 2, characterized in that The method of performing convolution analysis on the customer behavior image where the character area exists by using a dynamic behavior filter to obtain a behavior response graph of the customer behavior image, and calculating the character interest points of the customer behavior image based on the behavior response graph comprises the following steps: Inputting the customer dynamic video with the character area marked into the dynamic behavior filter for filtering; Performing convolution processing in different directions on each frame of the customer behavior image of the customer dynamic video that has completed the filtering processing, to obtain a horizontal response map and a vertical response map of each frame of the customer behavior image; Combining the horizontal response graphs and the vertical response graphs corresponding to all the customer behavior images to obtain behavior response graphs for all the customer behavior images; Calculate the behavior response values ​​of all behavior pixels in the behavior response graph using the response function; For each frame of the customer behavior image, respectively calculating the sum of the behavior saliency values ​​of all the behavior pixels in the person area and non-person area in the customer behavior image to obtain a regional saliency value and a non-regional saliency value, wherein the non-person area is an area other than the person area in the customer behavior image; According to all the region saliency values ​​and the non-region saliency values, weights are respectively assigned to the human region and the non-human region; The pixel threshold formula is used to assign thresholds to the behavior pixels in the character area and the non-character area after weight assignment to obtain the regional response threshold and the non-regional response threshold. The pixel threshold formula is as follows: Among them, D a is the regional response threshold, D b is the non-regional response threshold, is the regional saliency value of the customer behavior image corresponding to the i-th time node, is the non-regional saliency value of the customer behavior image corresponding to the i-th time node, ∈ is a preset fixed value, γ is a preset time period, β a and β b are the weights of the character area and the non-character area, t is the time coordinate of the customer behavior image corresponding to the behavior response graph, G a is the sum of all the regional saliency values ​​of all the customer behavior images, G b is the sum of all the non-regional saliency values ​​of all the customer behavior images; For each of the character regions in all the customer behavior images, the behavior pixel points in the character region whose behavior response value is greater than the region response threshold are screened out as character interest points of the customer behavior image; The behavior pixel points in the non-person area whose behavior response values ​​are greater than the non-area response threshold are screened out as the person interest points of the customer behavior image.

4. The method according to claim 3, characterized in that The response function is as follows: Wherein, H(x, y) is a two-dimensional Gaussian smoothing kernel, (x, y) is the pixel coordinate of the behavior pixel point on the behavior response map, σ, ω and τ are preset parameters of the dynamic behavior filter, and T is the behavior response map.

5. The method according to claim 2, characterized in that: The step of calculating the optical flow histograms of all the customer behavior images by using the optical flow estimation algorithm and integrating all the optical flow histograms to obtain the global behavior features of the target customer comprises the following steps: Extracting multiple frames of continuous customer behavior images from the customer dynamic video in time order as candidate behavior images; Calculate the optical flow vectors of the character interest points in all the candidate behavior images using an optical flow estimation algorithm; Calculate the optical flow amplitude and optical flow direction of all the character's interest points according to the optical flow vector of the character's interest point; constructing an optical flow histogram for all the candidate behavior images according to the optical flow amplitude and the optical flow direction; After traversing all the customer behavior images in the customer dynamic video, all the optical flow histograms are integrated to obtain the global behavior characteristics of the target customer.

6. The method according to claim 1, characterized in that The identifying the second abnormal behavior of the target customer within the information collection time period by combining the gravity sensor information and the image sensor information comprises the following steps: Extracting the gravity change characteristics of the gravity sensor information at the cut-off time node, and analyzing the gravity change trend of the intelligent unmanned vending machine according to the gravity change characteristics; If the gravity change trend is that gravity continues to increase, it is determined that the target customer has a second abnormal behavior during the information collection time period; If the gravity change trend is that the gravity continues to decrease, it is determined that the target customer does not have the second abnormal behavior during the information collection time period; If the gravity change trend is that the gravity fluctuates up and down, the gravity difference between the initial time node and the end time node is calculated, and the second abnormal behavior of the target customer during the information collection time period is identified by combining the gravity difference and the image sensor information.

7. The method according to claim 6, characterized in that The calculating of the gravity difference between the initial time node and the cut-off time node, and identifying the second abnormal behavior of the target customer within the information collection time period by combining the gravity difference and the image sensor information comprises the following steps: Acquire the initial gravity of the initial time node and the cutoff gravity of the cutoff time node according to the gravity sensor information; determining whether a gravity difference value obtained by subtracting the initial gravity from the cut-off gravity is zero, and if the gravity difference value is zero, identifying the second abnormal behavior of the target customer during the information collection time period through the image sensor information; If the gravity difference is not zero, analyzing whether the gravity difference is greater than zero; if the gravity difference is less than zero, identifying the second abnormal behavior of the target customer during the information collection time period through the image sensor information; If the gravity difference is greater than zero, it is determined that the target customer has the second abnormal behavior during the information collection time period.

8. The method according to claim 7, characterized in that The step of identifying the second abnormal behavior of the target customer within the information collection time period by using the image sensor information comprises the following steps: Using the image sensor information collected at the initial time node as the reference image information, and using the image sensor information collected at the cutoff time node as the target image information; Performing image registration on the reference image information and the target image information using an image registration technology; Extracting a color histogram of the reference image information and the target image information after image registration is completed; grayscale processing is performed on the reference image information and the target image information that have completed the registration; Using an edge detection algorithm to respectively extract the contours of the commodity in the reference image information and the target image information that have been grayed; Acquire shape features of the reference image information and the target image information according to the product outline; comparing the color histogram and the shape feature between the reference image information and the target image information respectively, and if the color histogram and the shape feature are the same, determining that the target customer does not have the second abnormal behavior during the information collection time period; If the color histogram and / or the shape feature are different, it is determined that the target customer has the second abnormal behavior during the information collection time period.

9. The method according to claim 8, characterized in that The method further comprises: If the target customer does not have the second abnormal behavior, the reference image information and the target image information are respectively input into a preset commodity recognition model to obtain reference commodity information and target commodity information; Comparing the reference commodity information with the target commodity information to obtain the types and quantities of commodities sold by the intelligent unmanned vending machine; Calculating the weight of the sold goods according to the types of the sold goods and the quantity of the sold goods; comparing the absolute value of the difference between the weight of the sold commodity and the gravity difference to see whether they are the same; if the weight of the sold commodity is not the same as the absolute value of the difference, inputting the reference commodity information and the target commodity information into the commodity recognition model again until the weight of the sold commodity is the same as the absolute value of the difference; If the weight of the sold goods is the same as the absolute value of the difference, outputting the bill information of the target customer according to the type of the sold goods and the quantity of the sold goods; If the target customer has the second abnormal behavior, the intelligent unmanned vending machine is controlled to output an alarm message.

10. An interactive behavior recognition system for an intelligent unmanned vending machine, characterized in that: include: a memory configured to store instructions; as well as A processor is configured to call the instructions from the memory and to implement the method for interactive behavior recognition for an intelligent unmanned vending machine according to any one of claims 1 to 9 when executing the instructions.

Images