Shopping behavior recognition method, system, device and medium for visual vending cabinets

Abstract

The present application discloses a method, system, device and medium for shopping behavior recognition for visual vending cabinets. The method obtains a video segment data set and video data to be recognized from the visual vending cabinet; uses a first algorithm to extract the coordinates of the goods in the video segment data set; uses a second algorithm to extract the coordinates of the hand joint points in the video segment data set; determines a time-space human-object relationship sequence diagram according to the coordinates of the goods and the coordinates of the hand joint points; uses the time-space human-object relationship sequence diagram to train a time-space graph convolutional network model to obtain a trained time-space graph convolutional network model; so as to use the trained time-space graph convolutional network model to classify the video data to be recognized, obtain atomic behavior results, and obtain a time-series composite shopping behavior recognition result based on the atomic behavior results. The present application can accurately identify whether the shopping behavior is normal and improve the detection speed of the network model.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a shopping behavior recognition method, system, device and medium for visual vending machines. Background Art

[0002] Smart vending machines are a new type of smart retail terminal equipment. As people have higher requirements for shopping flexibility and convenience, visual vending machines are not restricted by time and location, and can save manpower and complete transactions more conveniently and quickly. However, in many real consumption scenarios, there is still a common problem of unpredictable abnormal shopping behavior, and the advantages and value of smart technology cannot be effectively utilized in practice. Abnormal shopping behavior will not only cause economic losses, but may also affect food safety and endanger the health of consumers.

[0003] Nowadays, with the great progress of computer vision in various industries, the research on behavior recognition in the fields of public security, smart elderly care, etc. is increasing. In related research, there are relatively few methods for behavior recognition in the context of visual dynamic cabinets, and the behavior recognition advantages of hand skeleton data are not fully utilized, resulting in insufficient learning of key point data of the hand; at the same time, the previous graph convolutional network has problems such as large amount of calculation and slow detection speed, which leads to limited application in various fields and difficulty in achieving the expected results. Summary of the invention

[0004] The present application aims to solve at least one of the technical problems existing in the prior art. To this end, the present application proposes a shopping behavior recognition method, system, device and medium for visual vending machines, which can accurately identify whether the shopping behavior is normal and improve the detection speed of the network model.

[0005] In a first aspect, an embodiment of the present application provides a shopping behavior recognition method for a visual vending cabinet, and the shopping behavior recognition method for a visual vending cabinet includes:

[0006] Obtain a video segment dataset of a visual vending machine and video data to be identified;

[0007] Using a first algorithm to extract the coordinates of the products in the video segment data set;

[0008] Using a second algorithm to extract the coordinates of the hand joint points in the video segment data set;

[0009] Determine a time-space human-object relationship sequence diagram according to the commodity coordinates and the hand joint coordinates;

[0010] The spatiotemporal graph convolutional network model is trained by using the spatiotemporal human-object relationship sequence diagram to obtain a trained spatiotemporal graph convolutional network model; so that the trained spatiotemporal graph convolutional network model can be used to classify the video data to be identified to obtain atomic behavior results, and based on the atomic behavior results, a time-series composite shopping behavior recognition result is obtained.

[0011] Compared with the prior art, the first aspect of the present application has the following beneficial effects:

[0012] This method uses a first algorithm to extract the coordinates of goods in a video segment data set, and uses a second algorithm to extract the coordinates of hand joint points in the video segment data set. By extracting the coordinates of goods and hand joint points in the video segment data set, redundant information in the video is removed, and the detection speed of the spatiotemporal graph convolutional network model can be improved; then, a spatiotemporal person-object relationship sequence diagram is determined according to the coordinates of the goods and the coordinates of the hand joint points, and the spatiotemporal person-object relationship sequence diagram is used to train the spatiotemporal graph convolutional network model, so that the trained spatiotemporal graph convolutional network model can be used to classify the video data to be identified. By training the spatiotemporal graph convolutional network model using node features in two dimensions of time and space, the spatiotemporal graph convolutional network model can achieve atomic behavior classification with higher accuracy. Finally, a temporally composite shopping behavior recognition result is obtained according to the atomic behavior result, and whether the shopping behavior is normal can be accurately identified according to the shopping behavior recognition result.

[0013] According to some embodiments of the present application, extracting the coordinates of the products in the video segment dataset using a first algorithm includes:

[0014] Using a first algorithm to extract a rectangular frame of the product in the video segment data set;

[0015] The coordinates of the commodity are represented by the midpoint coordinates of the diagonal line of the rectangular frame.

[0016] According to some embodiments of the present application, the extracting the hand joint point coordinates in the video segment dataset using a second algorithm includes:

[0017] Using a second algorithm to extract the hand skeleton data in the video segment data set to obtain original hand joint point coordinates;

[0018] The missing coordinates in the original hand joint point coordinates are filled with data to obtain the final hand joint point coordinates.

[0019] According to some embodiments of the present application, determining the spatiotemporal human-object relationship sequence diagram according to the commodity coordinates and the hand joint point coordinates includes:

[0020] The product coordinates and the hand joint point coordinates are used as joint nodes and product nodes;

[0021] An undirected spatiotemporal graph is constructed using the joint nodes and the commodity nodes to obtain a spatiotemporal human-object relationship sequence graph.

[0022] According to some embodiments of the present application, before using the spatiotemporal human-object relationship sequence diagram to train the spatiotemporal graph convolutional network model, the shopping behavior recognition method for visual vending machines further includes:

[0023] A cross entropy loss function is used as the loss function of the spatiotemporal graph convolutional network model, and an SGD optimizer is used to perform stochastic gradient descent and training parameter update on the spatiotemporal graph convolutional network model to obtain a trained spatiotemporal graph convolutional network model.

[0024] According to some embodiments of the present application, the cross entropy loss function is used as the loss function of the spatiotemporal graph convolutional network model, and the SGD optimizer is used to perform stochastic gradient descent and training parameter update on the spatiotemporal graph convolutional network model to obtain a trained spatiotemporal graph convolutional network model, including:

[0025] The cross entropy loss function is used to back-propagate the error to each layer of the spatiotemporal graph convolutional network model, and the SGD optimizer is used to perform stochastic gradient descent and training parameter update on the spatiotemporal graph convolutional network model to adjust the weights and parameters of the network;

[0026] When the loss of each training iteration gradually decreases until the loss change between two adjacent iterations is less than the preset loss change threshold, the spatiotemporal graph convolutional network model converges and a trained spatiotemporal graph convolutional network model is obtained.

[0027] According to some embodiments of the present application, the stochastic gradient descent process is represented as follows:

[0028] θ t =θ t-1 -ηm t

[0029] m t =βm t-1 +(1-β)g t-1

[0030] Among them, θ t represents the training parameters at the tth iteration, θ t-1 represents the training parameters at the t-1th iteration, η represents the learning rate, m t represents the momentum parameter at the tth iteration, m t-1 represents the momentum parameter at the t-1th iteration, β represents the hyperparameter, and g t-1 Represents the gradient change rate at the t-1th iteration.

[0031] In a second aspect, the embodiment of the present application further provides a shopping behavior recognition system for a visual vending cabinet, and the shopping behavior recognition system for a visual vending cabinet includes:

[0032] A video data acquisition unit, used to acquire a video segment data set of a visual vending cabinet and video data to be identified;

[0033] A first data extraction unit, configured to extract the coordinates of the commodities in the video segment data set by using a first algorithm;

[0034] A second data extraction unit, configured to extract the coordinates of the hand joint points in the video segment data set by using a second algorithm;

[0035] A sequence diagram determining unit, used to determine a spatiotemporal human-object relationship sequence diagram according to the commodity coordinates and the hand joint point coordinates;

[0036] The shopping behavior recognition unit is used to train the spatiotemporal graph convolutional network model using the spatiotemporal person-object relationship sequence diagram to obtain a trained spatiotemporal graph convolutional network model; so as to use the trained spatiotemporal graph convolutional network model to classify the video data to be identified to obtain atomic behavior results, and obtain a time-series composite shopping behavior recognition result based on the atomic behavior results.

[0037] In the third aspect, an embodiment of the present application also provides a shopping behavior identification device for visual vending machines, comprising at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions that can be executed by the at least one control processor, and the instructions are executed by the at least one control processor so that the at least one control processor can execute a shopping behavior identification method for visual vending machines as described above.

[0038] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the above-mentioned shopping behavior recognition method for visual vending machines.

[0039] It can be understood that the beneficial effects of the second to fourth aspects compared with the related art are the same as the beneficial effects of the first aspect compared with the related art. Please refer to the relevant description in the first aspect, and no further details will be given here. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0041] Figure 1It is a flowchart of a method for identifying shopping behavior for a visual vending machine according to an embodiment of the present application;

[0042] Figure 2 This is a diagram of the natural structure of the human hand joints according to an embodiment of the present application;

[0043] Figure 3 is a schematic diagram of the ST-GCN network module of an embodiment of the present application;

[0044] Figure 4 It is a structural schematic diagram of a shopping behavior recognition system for visual vending cabinets according to an embodiment of the present application;

[0045] Figure 5 It is a structural schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0046] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as limiting the present application.

[0047] In the description of this application, if there is a description of first, second, etc., it is only for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0048] In the description of the present application, it should be understood that the descriptions involving orientation, such as the orientation or positional relationship indicated as up, down, etc., are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.

[0049] In the description of this application, it should be noted that, unless otherwise clearly defined, terms such as setting, installing, connecting, etc. should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meaning of the above terms in this application based on the specific content of the technical solution.

[0050] Since there are relatively few behavior recognition methods in the context of visual dynamic cabinets in related research, and the behavior recognition advantages of hand skeleton data have not been fully utilized, the learning of hand key point data is not sufficient; at the same time, the previous graph convolutional networks have problems such as large computational complexity and slow detection speed, which has limited their application in various fields and made it difficult to achieve the expected results.

[0051] To solve the above problems, the present application adopts a first algorithm to extract the coordinates of the goods in the video segment data set, and adopts a second algorithm to extract the coordinates of the hand joint points in the video segment data set. By extracting the coordinates of the goods and the coordinates of the hand joint points in the video segment data set, redundant information in the video is removed, and the detection speed of the spatiotemporal graph convolutional network model can be improved; then the spatiotemporal human-object relationship sequence diagram is determined according to the coordinates of the goods and the coordinates of the hand joint points, and the spatiotemporal human-object relationship sequence diagram is used to train the spatiotemporal graph convolutional network model, so that the trained spatiotemporal graph convolutional network model can be used to classify the video data to be identified. By training the spatiotemporal graph convolutional network model using the node features in the two dimensions of time and space, the spatiotemporal graph convolutional network model can achieve higher accuracy atomic behavior classification, and finally, a time-complex shopping behavior recognition result is obtained according to the atomic behavior result, so that whether the shopping behavior is normal can be accurately identified according to the shopping behavior recognition result.

[0052] Reference Figure 1 The embodiment of the present application provides a shopping behavior recognition method for a visual vending cabinet. The shopping behavior recognition method for a visual vending cabinet includes but is not limited to steps S100 to S500, wherein:

[0053] Step S100, obtaining a video segment data set of a visual vending cabinet and video data to be identified;

[0054] Step S200: extracting the coordinates of the products in the video segment data set using a first algorithm;

[0055] Step S300, extracting the coordinates of the hand joint points in the video segment data set using a second algorithm;

[0056] Step S400, determining a time-space human-object relationship sequence diagram according to the commodity coordinates and the hand joint coordinates;

[0057] Step S500, use the spatiotemporal human-object relationship sequence diagram to train the spatiotemporal graph convolutional network model to obtain a trained spatiotemporal graph convolutional network model; so that the trained spatiotemporal graph convolutional network model can be used to classify the video data to be identified, obtain atomic behavior results, and obtain a time-series composite shopping behavior recognition result based on the atomic behavior results.

[0058] In this embodiment, in order to improve the detection speed of the spatiotemporal graph convolutional network model, this embodiment obtains a video segment data set of a visual vending machine and video data to be identified, uses a first algorithm to extract the product coordinates in the video segment data set, and uses a second algorithm to extract the hand joint point coordinates in the video segment data set; in order to achieve higher accuracy atomic behavior classification of the spatiotemporal graph convolutional network model, this embodiment determines the spatiotemporal human-object relationship sequence diagram according to the product coordinates and the hand joint point coordinates, and uses the spatiotemporal human-object relationship sequence diagram to train the spatiotemporal graph convolutional network model to obtain a trained spatiotemporal graph convolutional network model; in order to accurately identify whether the shopping behavior is normal, this embodiment uses the trained spatiotemporal graph convolutional network model to classify the video data to be identified, obtains the atomic behavior results, and obtains the time-series composite shopping behavior recognition results based on the atomic behavior results.

[0059] In some embodiments, extracting the coordinates of the products in the video segment dataset using a first algorithm includes:

[0060] A first algorithm is used to extract a rectangular frame of the product in the video segment data set;

[0061] The coordinates of the product are represented by the midpoint coordinates of the diagonal of the rectangular frame.

[0062] In this embodiment, the first algorithm can use the YOLO algorithm, and the rectangular frame of the product in the video segment data set is extracted by the YOLO algorithm, and then the coordinates of the product are represented by the midpoint coordinates of the diagonal of the rectangular frame. In this embodiment, the YOLO algorithm can extract relatively accurate product coordinates, which provides a good data foundation for subsequent training models.

[0063] It should be noted that the first algorithm of this embodiment includes but is not limited to the YOLO algorithm, and this embodiment does not make any specific limitation.

[0064] In some embodiments, the second algorithm is used to extract the coordinates of the hand joint points in the video segment dataset, including:

[0065] The second algorithm is used to extract the hand skeleton data in the video segment data set to obtain the original hand joint point coordinates;

[0066] The missing coordinates in the original hand joint point coordinates are filled with data to obtain the final hand joint point coordinates.

[0067] In this embodiment, the second algorithm can use the OpenPose algorithm, and the OpenPose algorithm is used to extract the hand bone data in the video segment data set to obtain the original hand joint point coordinates, and then the missing coordinates in the original hand joint point coordinates are filled with data to obtain the final hand joint point coordinates. This embodiment can extract more accurate hand joint point coordinates through the OpenPose algorithm, and fill in the missing human hand joint point coordinates or human hand joint point coordinates with noise, which can provide a good data foundation for subsequent training models and avoid dimensional conflicts in the forward propagation of the model.

[0068] It should be noted that the second algorithm of this embodiment includes but is not limited to the OpenPose algorithm, and this embodiment does not make any specific limitation.

[0069] In some embodiments, determining a time-space human-object relationship sequence diagram based on the commodity coordinates and the hand joint coordinates includes:

[0070] The product coordinates and hand joint coordinates are used as joint nodes and product nodes;

[0071] An undirected spatiotemporal graph is constructed using joint nodes and commodity nodes to obtain a spatiotemporal human-object relationship sequence graph.

[0072] In this embodiment, an undirected spatiotemporal graph G = (V, E) is constructed on a human-object relationship sequence containing multiple nodes and multiple frames of data. The spatiotemporal human-object relationship sequence has all hand joint point coordinates and product coordinates, joint internal connections, and inter-frame connections. This embodiment constructs a spatiotemporal human-object relationship sequence graph for model training, which can utilize node features in both time and space dimensions, thereby providing a good data foundation for achieving accurate model classification.

[0073] In some embodiments, before using the spatiotemporal person-object relationship sequence graph to train the spatiotemporal graph convolutional network model, the shopping behavior recognition method for visual vending machines further includes:

[0074] The cross entropy loss function is used as the loss function of the spatiotemporal graph convolutional network model, and the SGD optimizer is used to perform stochastic gradient descent and update the training parameters of the spatiotemporal graph convolutional network model to obtain a trained spatiotemporal graph convolutional network model.

[0075] In this embodiment, the cross entropy loss function and the SGD optimizer are used to better train the spatiotemporal graph convolutional network model, thereby obtaining a spatiotemporal graph convolutional network model with high classification accuracy.

[0076] In some embodiments, a cross entropy loss function is used as the loss function of the spatiotemporal graph convolutional network model, and an SGD optimizer is used to perform stochastic gradient descent and training parameter updates on the spatiotemporal graph convolutional network model to obtain a trained spatiotemporal graph convolutional network model, including:

[0077] The cross entropy loss function is used to back-propagate the error to each layer of the spatiotemporal graph convolutional network model, and the SGD optimizer is used to perform stochastic gradient descent and training parameter updates on the spatiotemporal graph convolutional network model to adjust the weights and parameters of the network;

[0078] When the loss of each training iteration gradually decreases until the loss change between two adjacent iterations is less than the preset loss change threshold, the spatiotemporal graph convolutional network model converges and a trained spatiotemporal graph convolutional network model is obtained.

[0079] In this embodiment, the error is back-propagated to each layer of the spatiotemporal graph convolutional network model by using the cross entropy loss function, and the SGD optimizer is used to perform stochastic gradient descent and training parameter update on the spatiotemporal graph convolutional network model to adjust the weights and parameters of the network until the loss of each training iteration gradually decreases to the point where the loss change between two adjacent iterations is less than a preset loss change threshold. The iteration is then stopped to obtain a trained spatiotemporal graph convolutional network model with high classification accuracy.

[0080] In some embodiments, the stochastic gradient descent process is represented as follows:

[0081] θ t =θ t-1 -ηm t

[0082] m t =βm t-1 +(1-β)g t-1

[0083] Among them, θ t represents the training parameters at the tth iteration, θ t-1 represents the training parameters at the t-1th iteration, η represents the learning rate, m t represents the momentum parameter at the tth iteration, m t-1 represents the momentum parameter at the t-1th iteration, β represents the hyperparameter, and g t-1 Represents the gradient change rate at the t-1th iteration.

[0084] To facilitate understanding by those skilled in the art, a set of best embodiments is provided below:

[0085] In this embodiment, in view of the defects of the prior art, a behavior recognition method based on skeleton posture and spatiotemporal graph convolutional network is provided for the visual vending machine scene, which can utilize the information of time and space dimensions, integrate the coordinates of the commodity and the coordinates of the human hand joints for training, and realize a more accurate atomic behavior recognition method, and then obtain the shopping behavior recognition result of time series compound by encoding the atomic behavior. Specifically, it includes the following steps:

[0086] Step 1: Build a dataset containing shopping video segments in several visual vending machine scenarios.

[0087] In this embodiment, a camera is installed in the visual vending cabinet, and the angle of the camera lens is set to collect shopping videos, which include video data for network model training and video data to be identified. According to the different behaviors of consumers in the shopping cabinet, the video data obtained for network model training is manually segmented into several of the four actions of taking, twisting, drinking, and putting back, and then the video frames are extracted at a rate of 25 frames per second and put into the corresponding sub-datasets to form a video segment data set. The obtained video segment data set is divided into a training set, a test set, and a verification set; the training set is used to be sent to the spatiotemporal graph convolutional network model (i.e., the ST-GCN model) for training, the test set is used to be input into the trained ST-GCN model for testing, and the verification set is used to verify the test results.

[0088] Step 2: Use the YOLO algorithm to extract the coordinates of the product in the video frame.

[0089] The product coordinates (x, y) in the video frame are extracted. The rectangular frame of the product is first obtained through the YOLOv5 algorithm. Then, the midpoint coordinates of the diagonal of the rectangular frame are used to represent the product coordinates. The product coordinates can be expressed as:

[0090]

[0091] Among them, (x1, y1) represents the coordinates of the upper left corner of the rectangular box, and (x2, y2) represents the coordinates of the lower right corner of the rectangular box.

[0092] Step 3: Use the OpenPose algorithm to extract the hand skeleton data in the video frame to obtain the coordinates of the hand joint points.

[0093] like Figure 2As shown in the figure, it is a natural structure diagram of human hand joints; the hand bone data in the video frame is extracted through the OpenPose algorithm to obtain the original hand joint coordinates. The original hand joint coordinates are filled with data, and the missing bone key points are filled with 0. Since the human action video directly obtained through the camera may have problems such as occlusion and jitter, the human hand joint coordinates obtained by OpenPose will have certain noise, and even the missing of some hand joints will directly affect the results of behavior recognition. For the missing joints, fill them with 0 to avoid dimensional conflicts in the forward propagation of the model; each hand has 21 joint coordinates, and finally a total of 42 joint coordinates for both hands are obtained.

[0094] Step 4: Construct a time-space human-object relationship sequence diagram.

[0095] An undirected spatiotemporal graph (i.e., a spatiotemporal human-object relationship sequence graph) G = (V, E) is constructed on a human-object relationship sequence containing N nodes and T frames of data. The spatiotemporal human-object relationship sequence has internal connections within joints and inter-frame connections. The node set V includes all joints and commodity nodes in the joint point sequence of both hands, expressed as V = {v ti |t=1,...,T,i=1,...,N},v ti represents the i-th node in the t-th frame of the node data; the edge set E = {E S ,E F} contains two subsets. According to the connectivity of the human hand structure, connections are established between the joints in a single frame, and then the product nodes are connected to the wrist nodes to form the first subset E S = {v ti v tj |(i,j)∈H}, where H represents the natural connection relationship of the human hand joints and the set of connection relationships between the product and the wrists; then connect the same joints in consecutive frames as edges to form the second subset E F = {v ti v (t+1)i}, where all E of a particular joint i F The edges in will represent its trajectory over time.

[0096] Step 5: Send the constructed spatiotemporal human-object relationship sequence graph into the spatiotemporal graph convolutional network model for training.

[0097] The spatiotemporal human-object relationship sequence diagram is input into the spatiotemporal graph convolutional network model for training. The cross entropy loss function is used as the loss function of the model. The SGD optimizer is used for stochastic gradient descent and training parameter update. The spatiotemporal human-object relationship sequence diagram is input into the input end, and the prediction result is obtained from the output end. The error is back-propagated to each layer of the network according to the loss function, and the weights and parameters of the network are automatically adjusted. The loss of each training iteration gradually decreases until the loss of the training set and the loss of the test set are both stable. When the loss change between two iterations is less than the set loss change threshold, the model converges, the model training is completed, and the model weight is obtained. Specifically:

[0098] Constructing a spatiotemporal graph convolutional network, the network module diagram of ST-GCN is as follows Figure 3 As shown, except for the input layer, BN layer, Softmax layer and the final output classification layer, the remaining 9 modules are all ST-GCN blocks, where the step size of the black-gray ST-GCN block is set to 2, and the step size of the remaining ST-GCN blocks is set to 1. In order to prevent overfitting, after each ST-GCN block, dropout is set to 0.5. The input data is first normalized through the BN layer, and the output channels of each block in the middle are 64, 64, 64, 128, 128, 128, 256, 256, 256, and finally Softmax classification is performed to obtain the atomic behavior prediction results. In this embodiment, 70 training iterations are set, the initial learning rate is set to 0.01, and the learning rate is decayed at the 10th epoch and the 50th epoch.

[0099] The training of this spatiotemporal graph convolutional network model is an end-to-end process. The spatiotemporal human-object relationship sequence diagram is passed into the input end, and the prediction result is obtained from the output end. According to the loss function, the error is back-propagated to each layer of the network, and the weights and parameters of the network are automatically adjusted. The loss of each training iteration gradually decreases until the loss of the training set and the loss of the test set tend to be stable. When the loss change between two iterations is less than the set loss change value, it means that the model convergence condition has been met, and the model training can be completed to obtain the model weight. The network training uses the SGD optimizer for stochastic gradient descent, and uses the cross entropy loss as the loss function of the model to measure the difference between the predicted value and the actual value. The formula of the cross entropy loss function is as follows:

[0100]

[0101] Among them, x represents the true label, represents the distribution of the predicted classes, and n represents the number of spatiotemporal person-object relationship sequence graphs.

[0102] The formula for the gradient descent process of training parameters is as follows:

[0103] θt =θ t-1 -ηm t

[0104] m t =βm t-1 +(1-β)g t-1

[0105] Among them, θ t represents the training parameters at the tth iteration, θ t-1 represents the training parameters at the t-1th iteration, η represents the learning rate, m t represents the momentum parameter at the tth iteration, m t-1 represents the momentum parameter at the t-1th iteration, β represents the hyperparameter, and g t-1 Represents the gradient change rate at the t-1th iteration.

[0106] Step 6: Use the trained spatiotemporal graph convolutional network model to classify the collected video data and obtain the atomic behavior results.

[0107] The video collected in step 1 (i.e., the video data to be identified) is preprocessed, that is, the spatiotemporal human-object relationship sequence diagram of the preprocessed video is obtained in the order of steps 2, 3, and 4, and then sent to the trained spatiotemporal graph convolutional network model for behavior classification, and the atomic behavior recognition result is output.

[0108] Step 7: Encode the atomic behavior to obtain the temporal composite behavior recognition result.

[0109] The encoder is used to encode the input atomic behavior sequence, and after obtaining a coded sequence, it is handed over to the decoder. The decoder decodes the coded sequence and outputs a time-series composite shopping behavior recognition result.

[0110] It should be noted that time-complex shopping behavior refers to continuous behavior.

[0111] In this embodiment, the present application first performs video preprocessing on the video images under surveillance, and then extracts goods and hand joints from the processed video through YOLO and OpenPose algorithms to remove redundant information in the video, thereby improving the detection speed of the network model. The processed spatiotemporal human-object relationship sequence diagram is then sent to the spatiotemporal graph convolutional network ST-GCN model for feature extraction, and the node features in both time and space dimensions are used to achieve more accurate atomic behavior classification. The atomic behaviors are then encoded to obtain a time-complex shopping behavior recognition result to determine whether the consumer's shopping behavior is normal, which is well suited for the field of visual vending machines.

[0112] Reference Figure 4The embodiment of the present application further provides a shopping behavior recognition system for visual vending cabinets. The shopping behavior recognition system for visual vending cabinets includes a video data acquisition unit 100, a first data extraction unit 200, a second data extraction unit 300, a sequence diagram determination unit 400 and a shopping behavior recognition unit 500, wherein:

[0113] The video data acquisition unit 100 is used to acquire a video segment data set of a visual vending cabinet and video data to be identified;

[0114] The first data extraction unit 200 is used to extract the coordinates of the products in the video segment data set by using a first algorithm;

[0115] The second data extraction unit 300 is used to extract the coordinates of the hand joint points in the video segment data set using a second algorithm;

[0116] A sequence diagram determining unit 400 is used to determine a time-space human-object relationship sequence diagram according to the commodity coordinates and the hand joint point coordinates;

[0117] The shopping behavior recognition unit 500 is used to train the spatiotemporal graph convolutional network model using the spatiotemporal person-object relationship sequence diagram to obtain a trained spatiotemporal graph convolutional network model; so as to use the trained spatiotemporal graph convolutional network model to classify the video data to be recognized, obtain atomic behavior results, and obtain a time-series composite shopping behavior recognition result based on the atomic behavior results.

[0118] It should be noted that since the shopping behavior recognition system for visual vending machines in this embodiment and the above-mentioned shopping behavior recognition method for visual vending machines are based on the same inventive concept, the corresponding contents in the method embodiment are also applicable to the system embodiment and will not be described in detail here.

[0119] Reference Figure 5 The embodiment of the present application further provides a shopping behavior recognition device for a visual vending cabinet, and the shopping behavior recognition device for a visual vending cabinet includes:

[0120] at least one memory;

[0121] at least one processor;

[0122] at least one program;

[0123] The program is stored in the memory, and the processor executes at least one program to implement the above-mentioned shopping behavior recognition method for visual vending cabinets implemented in the present disclosure.

[0124] The electronic device may be any intelligent terminal including a mobile phone, a tablet computer, a personal digital assistant (PDA), a car computer, etc.

[0125] The electronic device according to the embodiment of the present application is described in detail below.

[0126] The processor 1600 may be implemented by a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided by the embodiments of the present disclosure;

[0127] The memory 1700 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1700 can store an operating system and other application programs. When the technical solution provided in the embodiments of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1700, and the processor 1600 calls and executes the shopping behavior recognition method for visual vending cabinets in the embodiment of the present disclosure.

[0128] Input / output interface 1800, used to implement information input and output;

[0129] The communication interface 1900 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0130] Bus 2000 , which transmits information between various components of the device (e.g., processor 1600 , memory 1700 , input / output interface 1800 , and communication interface 1900 );

[0131] The processor 1600 , the memory 1700 , the input / output interface 1800 , and the communication interface 1900 are connected to each other in communication within the device via the bus 2000 .

[0132] The embodiment of the present disclosure also provides a storage medium, which is a computer-readable storage medium. The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the above-mentioned shopping behavior recognition method for visual vending machines.

[0133] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0134] The embodiments described in the embodiments of the present disclosure are intended to more clearly illustrate the technical solutions of the embodiments of the present disclosure and do not constitute a limitation on the technical solutions provided by the embodiments of the present disclosure. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present disclosure are also applicable to similar technical problems.

[0135] Those skilled in the art will appreciate that the technical solutions shown in the figures do not limit the embodiments of the present disclosure and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0136] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0137] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0138] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0139] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0140] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0141] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0142] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0143] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable an electronic device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store programs. The above is a detailed description of the embodiments of the present application in conjunction with the accompanying drawings, but the present application is not limited to the above embodiments. Within the scope of knowledge possessed by ordinary technicians in the relevant technical field, various changes can be made without departing from the purpose of the present application.

[0144] The embodiments of the present application are described in detail above in conjunction with the accompanying drawings, but the present application is not limited to the above embodiments. Various changes can be made within the knowledge scope of ordinary technicians in the relevant technical field without departing from the purpose of the present application.

Claims

1. A shopping behavior recognition method for visual vending machines, characterized in that: The shopping behavior recognition method for visual vending cabinets includes: Obtain a video segment dataset of a visual vending machine and video data to be identified; Using a first algorithm to extract the coordinates of the products in the video segment data set; Using a second algorithm to extract the hand joint point coordinates in the video segment data set, the hand joint point coordinates include 21 joint point coordinates of each hand; According to the commodity coordinates and the hand joint coordinates, a time-space human-object relationship sequence diagram is determined; wherein: The product coordinates and the hand joint point coordinates are used as joint nodes and product nodes; The joint nodes and the commodity nodes are used to construct an undirected space-time graph to obtain a space-time human-object relationship sequence graph; specifically including: A spatiotemporal human-object relationship sequence graph is constructed on a human-object relationship sequence containing N nodes and T frames of data: G = (V, E). The spatiotemporal human-object relationship sequence has internal joint connections and inter-frame connections; the node set V includes all joints and commodity nodes in the hand joint point sequence, expressed as V = {v ti |t=1,...,T,i=1,...,N},v ti represents the i-th node in the t-th frame of the node data; the edge set E = {E S ,E F } contains two subsets. According to the connectivity of the human hand structure, connections are established between the joints in a single frame, and then the product nodes are connected to the wrist nodes to form the first subset E S = {v ti v tj |(i,j)∈H}, where H represents the natural connection relationship of the human hand joints and the set of connection relationships between the product and the wrists; then connect the same joints in consecutive frames as edges to form the second subset E F = {v ti v (t+1)i }, where all E of a particular joint i F The edges in E F Trajectory over time; The spatiotemporal graph convolutional network model is trained by using the spatiotemporal human-object relationship sequence diagram to obtain a trained spatiotemporal graph convolutional network model; so that the trained spatiotemporal graph convolutional network model can be used to classify the video data to be identified to obtain atomic behavior results, and based on the atomic behavior results, a time-series composite shopping behavior recognition result is obtained.

2. The method for identifying shopping behavior for visual vending machines according to claim 1, characterized in that: The extracting the commodity coordinates in the video segment data set by using the first algorithm includes: Using a first algorithm to extract a rectangular frame of the product in the video segment data set; The coordinates of the commodity are represented by the midpoint coordinates of the diagonal line of the rectangular frame.

3. The method for identifying shopping behavior for visual vending machines according to claim 1, characterized in that: The extracting the hand joint point coordinates in the video segment data set by using the second algorithm includes: Using a second algorithm to extract the hand skeleton data in the video segment data set to obtain original hand joint point coordinates; The missing coordinates in the original hand joint point coordinates are filled with data to obtain the final hand joint point coordinates.

4. The method for identifying shopping behavior for visual vending machines according to claim 1, characterized in that: Before using the spatiotemporal person-object relationship sequence graph to train the spatiotemporal graph convolutional network model, the shopping behavior recognition method for visual vending machines further includes: A cross entropy loss function is used as the loss function of the spatiotemporal graph convolutional network model, and an SGD optimizer is used to perform stochastic gradient descent and training parameter update on the spatiotemporal graph convolutional network model to obtain a trained spatiotemporal graph convolutional network model.

5. The method for identifying shopping behavior for visual vending machines according to claim 4, characterized in that: The cross entropy loss function is used as the loss function of the spatiotemporal graph convolutional network model, and the SGD optimizer is used to perform stochastic gradient descent and training parameter update on the spatiotemporal graph convolutional network model to obtain a trained spatiotemporal graph convolutional network model, including: The cross entropy loss function is used to back-propagate the error to each layer of the spatiotemporal graph convolutional network model, and the SGD optimizer is used to perform stochastic gradient descent and training parameter update on the spatiotemporal graph convolutional network model to adjust the weights and parameters of the network; When the loss of each training iteration gradually decreases until the loss change between two adjacent iterations is less than the preset loss change threshold, the spatiotemporal graph convolutional network model converges and a trained spatiotemporal graph convolutional network model is obtained.

6. The method for identifying shopping behavior for visual vending machines according to claim 5, characterized in that: The stochastic gradient descent process can be represented as follows: i t =θ t-1 -m t m t =βm t-1 +(1-β)g t-1 Among them, θ t represents the training parameters at the tth iteration, θ t-1 represents the training parameters at the t-1th iteration, η represents the learning rate, m t represents the momentum parameter at the tth iteration, m t-1 represents the momentum parameter at the t-1th iteration, β represents the hyperparameter, and g t-1 Represents the gradient change rate at the t-1th iteration.

7. A shopping behavior recognition system for visual vending machines, characterized in that: The shopping behavior recognition system for visual vending cabinets includes: A video data acquisition unit, used to acquire a video segment data set of a visual vending cabinet and video data to be identified; A first data extraction unit, configured to extract the coordinates of the commodities in the video segment data set by using a first algorithm; A second data extraction unit, configured to extract hand joint point coordinates in the video segment data set by using a second algorithm, wherein the hand joint point coordinates include 21 joint point coordinates of each hand; A sequence diagram determining unit is used to determine a time-space human-object relationship sequence diagram according to the commodity coordinates and the hand joint point coordinates; wherein: The product coordinates and the hand joint point coordinates are used as joint nodes and product nodes; The joint nodes and the commodity nodes are used to construct an undirected space-time graph to obtain a space-time human-object relationship sequence graph; specifically including: A spatiotemporal human-object relationship sequence graph is constructed on a human-object relationship sequence containing N nodes and T frames of data: G = (V, E). The spatiotemporal human-object relationship sequence has internal joint connections and inter-frame connections; the node set V includes all joints and commodity nodes in the hand joint point sequence, expressed as V = {v ti |t=1,...,T,i=1,...,N},v ti represents the i-th node in the t-th frame of the node data; the edge set E = {E S ,E F } contains two subsets. According to the connectivity of the human hand structure, connections are established between the joints in a single frame, and then the product nodes are connected to the wrist nodes to form the first subset E S = {v ti v tj |(i,j)∈H}, where H represents the natural connection relationship of the human hand joints and the set of connection relationships between the product and the wrists; then connect the same joints in consecutive frames as edges to form the second subset E F = {v ti v (t+1)i }, where all E of a particular joint i F The edges in E F Trajectory over time; The shopping behavior recognition unit is used to train the spatiotemporal graph convolutional network model using the spatiotemporal person-object relationship sequence diagram to obtain a trained spatiotemporal graph convolutional network model; so as to use the trained spatiotemporal graph convolutional network model to classify the video data to be identified to obtain atomic behavior results, and obtain a time-series composite shopping behavior recognition result based on the atomic behavior results.

8. A shopping behavior recognition device for visual vending cabinets, characterized in that: It includes at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions that can be executed by the at least one control processor, and the instructions are executed by the at least one control processor so that the at least one control processor can execute the shopping behavior recognition method for visual vending cabinets as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the shopping behavior recognition method for visual vending machines as described in any one of claims 1 to 6.

Images