Article tracking control method and system for intelligent storage cabinet

Abstract

The embodiment of the invention provides an article tracking control method and system for an intelligent storage cabinet, and the method comprises the steps: obtaining a feature image shot by an upper binocular camera of the intelligent storage cabinet; calling a preset emotion prediction model according to the feature image to obtain emotion trend data of the user, optimizing and iterating the weight parameters through a loss function, and inputting new human face limb features and voice features into the trained emotion prediction model; obtaining preliminary intention features of the article through the emotion trend data; performing article identification on the feature image to obtain initial position information of an article; and combining the preliminary intention features of the article with the initial position information of the article, calculating to obtain predicted trajectory information of the article, taking a user limb action result as an auxiliary consideration factor, and combining a depth image of article identification, so that a high-accuracy and rapid article tracking scheme is realized, the article tracking cost is reduced, complex and expensive hardware does not need to be set, and the implementation is convenient. And manpower resources are saved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to an item tracking control method of an intelligent storage cabinet, an item tracking control system of an intelligent storage cabinet, a computer device and a storage medium. Background Art

[0002] The existing item tracking methods for various storage cabinets are generally set on items or cabinet spaces through RFID, weight / pressure sensors, cameras, and Internet of Things (IoT) sensors. For example, vending storage cabinets can track items in the storage cabinets through RFID and weight / pressure sensors, but the cost of RFID and sensor hardware is relatively high. In addition, the commonly used item tracking method for existing vending storage cabinets is to use cameras to identify payment for items, which is relatively low in cost, but the accuracy of item tracking needs to be improved. Summary of the invention

[0003] In view of the above problems, embodiments of the present invention are proposed to provide an item tracking control method for a smart storage cabinet, an item tracking control system for a smart storage cabinet, a computer device and a storage medium that overcome the above problems or at least partially solve the above problems.

[0004] In order to solve the above problems, an embodiment of the present invention discloses an item tracking control method of a smart storage cabinet, comprising: Acquire the feature image captured by the binocular camera on the smart storage cabinet; Calling a preset emotion prediction model according to the feature image to obtain the user's emotion trend data; Obtaining preliminary intention characteristics of the item through the sentiment trend data; Performing object recognition on the feature image to obtain initial position information of the object; The preliminary intention features of the item and the initial location information of the item are combined to calculate the predicted trajectory information of the item.

[0005] Preferably, the method of calling a preset emotion prediction model according to the feature image to obtain the user's emotion trend data includes:

[0006] Perform voice, face or body recognition on the feature image to obtain face and body features; Acquire audio information corresponding to the feature image, extract speech features from the audio information, The facial and body features and voice features are input into the emotion prediction model to obtain the user's emotion trend data.

[0007] Preferably, the emotional trend data includes first emotional trend data and second emotional trend data; and obtaining the preliminary intention characteristics of the item through the emotional trend data includes: Determine the first bypass subtree and the final bypass subtree of the decision tree; The first sentiment trend data and the second sentiment trend data are respectively set as the characteristic attribute values ​​of the first bypass subtree and the final bypass subtree of the decision tree, to obtain the classification result of the output of the decision tree, and the classification result is determined as the preliminary intention feature of the item.

[0008] Preferably, the performing object recognition on the feature image to obtain the initial position information of the object includes: The feature image is input into a deep image recognition model to obtain output information of the initial position of the object.

[0009] Preferably, the combining of the preliminary intention feature of the item and the initial location information of the item to calculate the predicted trajectory information of the item includes: If the preliminary intention characteristic of the item is a positive trend, the first probability of the item moving to the first preset location is adjusted; If the preliminary intention characteristic of the item is a negative trend, then adjusting the second probability of the item moving to the second preset location; Generate object prediction trajectory information according to the first probability and the second probability.

[0010] Preferably, the facial features and voice features are input into the emotion prediction model to obtain the user's emotion trend data, including

[0011] Combining the facial and body features and voice features into feature training samples; Initializing the learning rate and weight parameters of the emotion prediction model; The aspect ratio between each image pixel in the feature training sample is calculated and normalized using an activation function to obtain the spatial relationship similarity; The feature similarity between each node is calculated by the linearly transformed image pixels in each node; The feature similarity between each speech feature point is calculated by the speech syllable after linear transformation in each speech feature; The specific target similarity between each node contained in the specific target area is obtained by calculating the image pixels of the specific target area; The matrix of node edge weights is constructed through the spatial relationship similarity, specific target similarity and feature similarity between each node in the sentiment prediction model; Constructing a fully connected graph of each node consisting of feature graph nodes of feature training samples, edges connecting nodes, and a matrix of node edge weights; The weight parameters are iterated through loss function optimization to obtain a trained emotion prediction model, and then new facial and body features and voice features are input into the trained emotion prediction model to obtain output emotion trend data.

[0012] The embodiment of the present invention discloses an item tracking control system of a smart storage cabinet, comprising: A first acquisition module is used to acquire a feature image captured by a binocular camera on the smart storage cabinet; An emotion trend data module, used to call a preset emotion prediction model according to the feature image to obtain the user's emotion trend data; A first recognition module, used to obtain preliminary intention features of an item through the sentiment trend data; A second recognition module is used to perform object recognition on the feature image to obtain initial position information of the object; The item prediction trajectory information module is used to combine the item's preliminary intention features and the item's initial position information to calculate the item's prediction trajectory information.

[0013] Preferably, the emotional trend data module includes

[0014] A face and body recognition submodule, used for performing voice, face or body recognition on the feature image to obtain face and body features; The audio submodule is used to obtain the audio information corresponding to the feature image and extract the speech features from the audio information. The input submodule is used to input the facial and body features and voice features into the emotion prediction model to obtain the user's emotion trend data.

[0015] An embodiment of the present invention further discloses a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of item tracking control of the above-mentioned smart storage cabinet when executing the computer program.

[0016] The embodiment of the present invention further discloses a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of item tracking control of the above-mentioned smart storage cabinet are implemented.

[0017] The embodiments of the present invention include the following advantages: In an embodiment of the present invention, the item tracking control method of the smart storage cabinet includes: acquiring a feature image captured by a binocular camera on the smart storage cabinet; calling a preset emotion prediction model according to the feature image to obtain the user's emotion trend data; obtaining a preliminary intention feature of the item through the emotion trend data; performing item identification on the feature image to obtain the item's initial position information; combining the preliminary intention feature of the item with the item's initial position information to calculate the item's predicted trajectory information, taking the user's body movement results as an auxiliary consideration factor, and combining the depth image of the item identification to achieve a high-accuracy and fast item tracking solution, thereby reducing the cost of item tracking, eliminating the need to set up complex and expensive hardware, and saving human resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 is a flowchart of steps of an embodiment of an item tracking control method of a smart storage cabinet according to an embodiment of the present invention; Figure 2 It is a structural block diagram of an embodiment of an item tracking control system of an intelligent storage cabinet according to an embodiment of the present invention; Figure 3 The diagram is a diagram of the internal structure of a computer device according to an embodiment. DETAILED DESCRIPTION

[0020] In order to make the technical problems, technical solutions and beneficial effects solved by the embodiments of the present invention more clearly understood, the embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0021] In an embodiment of the present invention, a binocular camera is provided on the smart storage cabinet, and the binocular camera can obtain a depth image of an item to be tracked and a depth image of a user, obtain the user's intended action based on the user's depth image, and then obtain the name of the item and other category information based on the depth image of the item, and combine the user's intended action with the name of the item and other category information to predict the location of the item, and use the user's body movement results as an auxiliary consideration combined with the depth image of the item recognition to implement a high-accuracy and fast item tracking solution, thereby reducing the cost of item tracking, eliminating the need to set up complex and expensive hardware, and saving human resources.

[0022] Reference Figure 1, shows a flowchart of an embodiment of an item tracking control method of a smart storage cabinet according to an embodiment of the present invention, which may specifically include the following steps: Step 101, obtaining a feature image captured by a binocular camera on the smart storage cabinet; In an embodiment of the present invention, the smart storage cabinet may be provided with a terminal and a binocular camera; the terminal may be, but is not limited to, various vehicle-mounted computers, personal computers, laptop computers, smart phones, tablet computers, and portable wearable devices. The embodiment of the present invention does not limit the specific type of the terminal, and the operating system of the terminal may include Android, Harmony OS, IOS, Windows Phone, Windows, etc., and the present invention does not impose excessive restrictions on this.

[0023] In a preferred embodiment, the item tracking control method of the smart storage cabinet provided by the embodiment of the present invention can be applied to an application environment including a terminal and a server. The terminal communicates with the server through a network. The terminal can be, but is not limited to, various personal computers, laptops, and tablet computers, and the server can be implemented as an independent server or a server cluster composed of multiple servers.

[0024] It should be noted that the smart storage cabinet may refer to a vending cabinet or a locker, and the embodiments of the present invention do not impose too many restrictions on this. In addition, with regard to the structure of the smart storage cabinet, the structure of the main cabinet may include side panels, top panels, bottom panels, back panels, supporting legs, bases, adjustable casters, etc.; the internal structure may include adjustable shelves or fixed shelves, vertical or horizontal partition dividers, sliding drawers, classification grids, pull-out metal mesh baskets, swing doors or sliding doors and corresponding mechanical locks, etc. The embodiments of the present invention do not impose too many restrictions on the structure and function of the smart storage cabinet.

[0025] Specifically in the embodiment of the present invention, the binocular camera refers to a camera that simultaneously collects images at two different positions to obtain a fused depth image, thereby realizing the binocular imaging function and solving the problem that users and objects may block each other. When the user opens the smart storage cabinet to take out something, the terminal can trigger the binocular camera to perform image acquisition operations to obtain feature images. Specifically, the smart storage cabinet can be provided with an opening and closing sensor between the cabinet door and the main cabinet body. When the user opens the cabinet door, the opening and closing sensor is triggered to send a signal to the terminal on the smart storage cabinet, and the terminal triggers the binocular camera to perform image acquisition operations. That is, the terminal can be connected to the opening and closing sensor and the binocular camera. It is worth noting that the opening and closing sensor can be a Hall sensor, and the embodiments of the present invention do not impose too many restrictions on this. Specifically, the feature image refers to an image that includes both a user image and an object image. After obtaining the feature image, the terminal can call an internal intelligent model to process the feature image data. The specific steps are as follows.

[0026] Step 102, calling a preset emotion prediction model according to the feature image to obtain the user's emotion trend data; In the embodiment of the present invention, the terminal may include a preset emotion prediction model, and after obtaining the feature image, the preset emotion prediction model is called to obtain the emotion trend data of the user.

[0027] The emotion prediction model can be composed of multiple neural networks. For example, the emotion prediction model can include three network parts. The first network part can include multiple hidden layers and activation layers connected in sequence to identify facial features and body features in the feature image; the second network part is used to extract voice features, which includes a voice conversion layer, a hidden layer, a fully connected layer, and a maximum pooling layer. Specifically, the voice conversion layer can include an encoder (Bi-LSTM) module, an attention module, and a decoder (LSTM) module. The embodiment of the present invention does not impose too many restrictions on the specific structure of the voice conversion layer; in addition, the third network part can include a fully connected graph module of each node, a hidden layer, a fully connected layer, a normalization layer, etc.

[0028] It is worth noting that the smart storage cabinet can be provided with a voice collection device, which can be used to collect the user's voice data; specifically, the voice collection device can also be connected to the terminal, and the terminal triggers and controls the voice collection device to perform voice collection.

[0029] Specifically applied to the embodiment of the present invention, the preset emotion prediction model is called according to the feature image to obtain the user's emotion trend data, including

[0030] Performing face or body recognition on the feature image to obtain face and body features; Acquire audio information corresponding to the feature image, extract speech features from the audio information, The facial and body features and voice features are input into the emotion prediction model to obtain the user's emotion trend data.

[0031] First, face or body recognition is performed on the feature image to obtain face and body features. Specifically, the feature image is input into the first network part to obtain output face and body features. Further, when the user's audio data information is detected, that is, the voice acquisition device collects the voice data of the user when taking an object, the voice features can also be extracted from the voice data. Specifically, the voice data is input into the second network part, and is recognized through the encoder (Bi-LSTM) module, the attention module, the decoder (LSTM) module, the hidden layer, the fully connected layer, and the maximum pooling layer to obtain output voice features. Then, the face and body features and the voice features are input into the emotion prediction model to obtain the user's emotion trend data. Specifically, the emotional trend data can be score data. For example, in the score range of [1-100], 1 represents the most positive trend and 100 represents the most negative trend. The score range of 1 to 100 represents the distribution process of transition from positive trends to negative trends. Negative trends can be trends expressed by facial or body features such as "fear" and "panic", while positive trends can be trends expressed by facial or body features such as "smile" and "nod".

[0032] In the embodiment of the present invention, by combining voice and facial or body movements and obtaining training samples through multimodal fusion, the robustness of model training can be enhanced, the accuracy of model recognition can be improved, and the real-time performance of model recognition can be improved.

[0033] Specifically applied to the embodiment of the present invention, the facial and body features and voice features are input into the emotion prediction model to obtain the user's emotion trend data, including

[0034] Combining the facial and body features and voice features into feature training samples; Initializing the learning rate and weight parameters of the emotion prediction model; The aspect ratio between image pixels in the feature training samples is calculated and normalized using an activation function to obtain the spatial relationship similarity; It should be noted that the aspect ratio refers to the ratio of the width to the height of the feature map in the feature training sample; the spatial relationship similarity is obtained by normalizing the aspect ratio; Specifically, the spatial relationship similarity S1 may include: ; in, Represents the spatial relationship similarity, represents the aspect ratio of the image pixels, i=1,2,3······n, where n is the number of image pixels, represents the first learnable parameter optimized by the first network part, Represents a second learnable parameter optimized for the first network portion.

[0035] The feature similarity between each speech feature point is calculated by the speech syllable after linear transformation in each speech feature; Among them, the feature similarity is: ; in, represents feature similarity, represents the first speech syllable component after linear transformation, i=1,2,3······N, N is the number of speech syllables, Represents the second speech syllable component after linear transformation, i=1,2,3······N, where N is the number of speech syllables.

[0036] The specific target similarity between each node in the specific target area is calculated by the image pixels of the specific target area; wherein the node refers to a pixel point in the feature map; and the specific target area refers to the intersection edge area of ​​the user's body part and the object; Furthermore, the specific target similarity can be expressed as follows: ; in, Indicates the specific target similarity, Indicates the ratio parameter of a specific target area to the feature map, Represents the width data of a specific target area, i=1,2,3······o, where o refers to the number of image pixels in the specific target area, Represents the height data of a specific target area, i=1,2,3······o, where o refers to the number of image pixels of the specific target area.

[0037] The matrix of node edge weights is constructed through the spatial relationship similarity, specific target similarity and feature similarity between each node in the sentiment prediction model; Then the matrix of the edge weight of this node is ; Constructing a fully connected graph of each node consisting of feature graph nodes of feature training samples, edges connecting nodes, and a matrix of node edge weights; The weight parameters are iterated through loss function optimization to obtain a trained emotion prediction model, and then new facial and body features and voice features are input into the trained emotion prediction model to obtain output emotion trend data.

[0038] Specifically, the loss function includes the following expression: ; in, represents the loss function, w represents the node edge weight, Represents the output emotional trend data, i=1,2,3······n, n is the number of image pixels, Represents the label data of the sample, i=1,2,3······n, n is the number of image pixels, It represents the ratio parameter of a specific target area to the feature map.

[0039] Step 103, obtaining preliminary intention characteristics of the item through the sentiment trend data; In the embodiment of the present invention, after obtaining the emotional trend data, the preliminary intention characteristics of the item can be obtained through the emotional trend data. Specifically, the emotional trend data includes the first emotional trend data and the second emotional trend data. The preliminary intention characteristics of the item obtained through the emotional trend data include: Determine the first bypass subtree and the final bypass subtree of the decision tree; The first sentiment trend data and the second sentiment trend data are respectively set as the characteristic attribute values ​​of the first bypass subtree and the final bypass subtree of the decision tree, to obtain the classification result of the output of the decision tree, and the classification result is determined as the preliminary intention feature of the item.

[0040] Specifically, the first emotional trend data may refer to positive trend data, and the second emotional trend data may refer to negative trend data. The positive trend data and the negative trend data are set in the subtrees on both sides of the edge of the decision tree structure. By combining the emotional prediction model with the decision tree, the accuracy of the user's intention prediction can be improved, the data preprocessing requirements are reduced, and the problem of easy overfitting of the model is solved. Step 104, performing object recognition on the feature image to obtain initial position information of the object; In a preferred embodiment of the embodiments of the present invention, performing object recognition on the feature image to obtain the initial position information of the object includes: inputting the feature image into a deep image recognition model to obtain output initial position information of the object.

[0041] The initial position information of the object may refer to three-dimensional coordinates of the x-axis, y-axis and z-axis, or may be coordinate data including depth distance information, and the embodiment of the present invention does not impose excessive restrictions on this.

[0042] It should be noted that the initial position information of the object obtained by using a binocular camera and a depth image recognition model can be obtained by a technician in this field through conventional technology, and the embodiments of the present invention will not be described in detail here.

[0043] Step 105, combining the preliminary intention features of the item and the initial location information of the item to calculate and obtain the predicted trajectory information of the item.

[0044] Specifically applied to the embodiments of the present invention, after obtaining the preliminary intention feature of the item and the initial position information of the item, the preliminary intention feature of the item and the initial position information of the item can be combined to generate the predicted trajectory information of the item; specifically, the preliminary intention feature of the item and the initial position information of the item are combined to calculate the predicted trajectory information of the item, including: if the preliminary intention feature of the item is a positive trend, adjusting the first probability of the item moving to the first preset location; if the preliminary intention feature of the item is a negative trend, adjusting the second probability of the item moving to the second preset location; generating the predicted trajectory information of the item with the initial position information of the item as the starting point according to the first probability and the second probability.

[0045] Specifically, the first preset location is the payment counter, and the first probability refers to the probability that the user walks to the payment counter with the item and "checks out immediately". The second preset location is the door, and the first probability refers to the probability that the user walks to the door with the item and "escapes the bill". The predicted trajectory of the item is generated by adjusting the probabilities of the two.

[0046] In an embodiment of the present invention, the item tracking control method of the smart storage cabinet includes: acquiring a feature image captured by a binocular camera on the smart storage cabinet; calling a preset emotion prediction model according to the feature image to obtain the user's emotion trend data; obtaining a preliminary intention feature of the item through the emotion trend data; performing item identification on the feature image to obtain the item's initial position information; combining the preliminary intention feature of the item with the item's initial position information to calculate the item's predicted trajectory information, taking the user's body movement results as an auxiliary consideration factor, and combining the depth image of the item identification to achieve a high-accuracy and fast item tracking solution, thereby reducing the cost of item tracking, eliminating the need to set up complex and expensive hardware, and saving human resources.

[0047] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0048] Reference Figure 2 , shows a structural block diagram of an embodiment of an item tracking control system of a smart storage cabinet according to an embodiment of the present invention, which may specifically include the following modules: The first acquisition module 301 is used to acquire a feature image captured by a binocular camera on the smart storage cabinet; The emotion trend data module 302 is used to call a preset emotion prediction model according to the feature image to obtain the emotion trend data of the user; A first recognition module 303, used to obtain preliminary intention features of items through the sentiment trend data; The second recognition module 304 is used to perform object recognition on the feature image to obtain initial position information of the object; The item prediction trajectory information module 305 is used to combine the item's preliminary intention features and the item's initial position information to calculate the item's prediction trajectory information.

[0049] Preferably, the emotional trend data module includes

[0050] A face and body recognition submodule, used for performing voice, face or body recognition on the feature image to obtain face and body features; The audio submodule is used to obtain the audio information corresponding to the feature image and extract the speech features from the audio information. The input submodule is used to input the facial and body features and voice features into the emotion prediction model to obtain the user's emotion trend data.

[0051] Preferably, the emotional trend data includes first emotional trend data and second emotional trend data; the first recognition module obtains preliminary intention features of the item through the emotional trend data including: A determination submodule, used to determine a first bypass subtree and a final bypass subtree of the decision tree; A submodule is set to set the first emotion trend data and the second emotion trend data as characteristic attribute values ​​of the first bypass subtree and the final bypass subtree of the decision tree respectively, obtain the classification result of the output of the decision tree, and determine the classification result as the preliminary intention feature of the item.

[0052] Preferably, the second identification module includes: The object initial position submodule is used to input the feature image into the deep image recognition model to obtain output object initial position information.

[0053] Preferably, the item prediction trajectory information module includes: A first probability submodule, for adjusting a first probability of the object moving to a first preset location if the preliminary intention feature of the object is a positive trend; A second probability submodule, for adjusting a second probability of the object moving to a second preset location if the preliminary intention characteristic of the object is a negative trend; The generation submodule is used to generate object prediction trajectory information according to the first probability and the second probability.

[0054] Preferably, the input submodule includes

[0055] A composition unit, used for composing the facial and body features and voice features into feature training samples; An initialization unit, used to initialize the learning rate and weight parameters of the emotion prediction model; Aspect ratio unit, used to calculate the aspect ratio between each image pixel in the feature training sample, and normalize it using an activation function to obtain the spatial relationship similarity; A first calculation unit, used for calculating the feature similarity between each node through the linearly transformed image pixels in each node; A second calculation unit is used to calculate the feature similarity between each speech feature point through the speech syllable after linear transformation in each speech feature; A third calculation unit is used to calculate the specific target similarity between each node included in the specific target area through the image pixels of the specific target area; A construction unit, used to construct a matrix of node edge weights through spatial relationship similarity, specific target similarity and feature similarity between each node in the sentiment prediction model; A fully connected graph unit, used to construct a fully connected graph of each node consisting of a feature graph node of a feature training sample, a matrix of connecting node edges and node edge weights; The iterative unit is used to optimize and iterate the weight parameters through a loss function to obtain a trained emotion prediction model, and then input new facial and body features and voice features into the trained emotion prediction model to obtain output emotion trend data.

[0056] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0057] For the specific definition of the item tracking control system of the smart storage cabinet, please refer to the definition of the item tracking control method of the smart storage cabinet above, which will not be repeated here. Each module in the above-mentioned item tracking control system of the smart storage cabinet can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0058] The item tracking control system of the smart storage cabinet provided above can be used to execute the item tracking control method of the smart storage cabinet provided in any of the above embodiments, and has corresponding functions and beneficial effects.

[0059] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 3 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for tracking and controlling items in an intelligent storage cabinet is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0060] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0061] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented: Acquire the feature image captured by the binocular camera on the smart storage cabinet; Calling a preset emotion prediction model according to the feature image to obtain the user's emotion trend data; Obtaining preliminary intention characteristics of the item through the sentiment trend data; Performing object recognition on the feature image to obtain initial position information of the object; The preliminary intention features of the item and the initial location information of the item are combined to calculate the predicted trajectory information of the item.

[0062] Preferably, the method of calling a preset emotion prediction model according to the feature image to obtain the user's emotion trend data includes:

[0063] Perform voice, face or body recognition on the feature image to obtain face and body features; Acquire audio information corresponding to the feature image, extract speech features from the audio information, The facial and body features and voice features are input into the emotion prediction model to obtain the user's emotion trend data.

[0064] Preferably, the emotional trend data includes first emotional trend data and second emotional trend data; and obtaining the preliminary intention characteristics of the item through the emotional trend data includes: Determine the first bypass subtree and the final bypass subtree of the decision tree; The first sentiment trend data and the second sentiment trend data are respectively set as the characteristic attribute values ​​of the first bypass subtree and the final bypass subtree of the decision tree, to obtain the classification result of the output of the decision tree, and the classification result is determined as the preliminary intention feature of the item.

[0065] Preferably, the performing object recognition on the feature image to obtain the initial position information of the object includes: The feature image is input into a deep image recognition model to obtain output information of the initial position of the object.

[0066] Preferably, the combining of the preliminary intention feature of the item and the initial location information of the item to calculate the predicted trajectory information of the item includes: If the preliminary intention characteristic of the item is a positive trend, the first probability of the item moving to the first preset location is adjusted; If the preliminary intention characteristic of the item is a negative trend, then adjusting the second probability of the item moving to the second preset location; Generate object prediction trajectory information according to the first probability and the second probability.

[0067] Preferably, the facial features and voice features are input into the emotion prediction model to obtain the user's emotion trend data, including

[0068] Combining the facial and body features and voice features into feature training samples; Initializing the learning rate and weight parameters of the emotion prediction model; The aspect ratio between each image pixel in the feature training sample is calculated and normalized using an activation function to obtain the spatial relationship similarity; The feature similarity between each node is calculated by the linearly transformed image pixels in each node; The feature similarity between each speech feature point is calculated by the speech syllable after linear transformation in each speech feature; The specific target similarity between each node contained in the specific target area is obtained by calculating the image pixels of the specific target area; The matrix of node edge weights is constructed through the spatial relationship similarity, specific target similarity and feature similarity between each node in the sentiment prediction model; Constructing a fully connected graph of each node consisting of feature graph nodes of feature training samples, edges connecting nodes, and a matrix of node edge weights; The weight parameters are iterated through loss function optimization to obtain a trained emotion prediction model, and then new facial and body features and voice features are input into the trained emotion prediction model to obtain output emotion trend data.

[0069] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented: Acquire the feature image captured by the binocular camera on the smart storage cabinet; Calling a preset emotion prediction model according to the feature image to obtain the user's emotion trend data; Obtaining preliminary intention characteristics of the item through the sentiment trend data; Performing object recognition on the feature image to obtain initial position information of the object; The preliminary intention features of the item and the initial location information of the item are combined to calculate the predicted trajectory information of the item.

[0070] Preferably, the method of calling a preset emotion prediction model according to the feature image to obtain the user's emotion trend data includes:

[0071] Perform voice, face or body recognition on the feature image to obtain face and body features; Acquire audio information corresponding to the feature image, extract speech features from the audio information, The facial and body features and voice features are input into the emotion prediction model to obtain the user's emotion trend data.

[0072] Preferably, the emotional trend data includes first emotional trend data and second emotional trend data; and obtaining the preliminary intention characteristics of the item through the emotional trend data includes: Determine the first bypass subtree and the final bypass subtree of the decision tree; The first sentiment trend data and the second sentiment trend data are respectively set as the characteristic attribute values ​​of the first bypass subtree and the final bypass subtree of the decision tree, to obtain the classification result of the output of the decision tree, and the classification result is determined as the preliminary intention feature of the item.

[0073] Preferably, the performing object recognition on the feature image to obtain the initial position information of the object includes: The feature image is input into a deep image recognition model to obtain output information of the initial position of the object.

[0074] Preferably, the combining of the preliminary intention feature of the item and the initial location information of the item to calculate the predicted trajectory information of the item includes: If the preliminary intention characteristic of the item is a positive trend, the first probability of the item moving to the first preset location is adjusted; If the preliminary intention characteristic of the item is a negative trend, then adjusting the second probability of the item moving to the second preset location; Generate object prediction trajectory information according to the first probability and the second probability.

[0075] Preferably, the facial features and voice features are input into the emotion prediction model to obtain the user's emotion trend data, including

[0076] Combining the facial and body features and voice features into feature training samples; Initializing the learning rate and weight parameters of the emotion prediction model; The aspect ratio between each image pixel in the feature training sample is calculated and normalized using an activation function to obtain the spatial relationship similarity; The feature similarity between each node is calculated by the linearly transformed image pixels in each node; The feature similarity between each speech feature point is calculated by the speech syllable after linear transformation in each speech feature; The specific target similarity between each node contained in the specific target area is obtained by calculating the image pixels of the specific target area; The matrix of node edge weights is constructed through the spatial relationship similarity, specific target similarity and feature similarity between each node in the sentiment prediction model; Constructing a fully connected graph of each node consisting of feature graph nodes of feature training samples, edges connecting nodes, and a matrix of node edge weights; The weight parameters are iterated through loss function optimization to obtain a trained emotion prediction model, and then new facial and body features and voice features are input into the trained emotion prediction model to obtain output emotion trend data.

[0077] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0078] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0079] It will be appreciated by those skilled in the art that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0080] The embodiments of the present invention are described with reference to flowcharts and / or block diagrams of apparatuses, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0081] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction method, which is implemented in the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0082] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0083] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0084] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, device, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, device, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a..." do not exclude the existence of other identical elements in the process, device, article or terminal device including the elements.

[0085] The above is a detailed introduction to an item tracking control method for an intelligent storage cabinet, an item tracking control system for an intelligent storage cabinet, a computer device and a storage medium provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for tracking and controlling items in a smart storage cabinet, characterized in that: include: Acquire the feature image captured by the binocular camera on the smart storage cabinet; Calling a preset emotion prediction model according to the feature image to obtain the user's emotion trend data; Obtaining preliminary intention characteristics of the item through the sentiment trend data; Performing object recognition on the feature image to obtain initial position information of the object; The preliminary intention features of the item and the initial location information of the item are combined to calculate the predicted trajectory information of the item.

2. The method according to claim 1, characterized in that The method of calling a preset emotion prediction model according to the feature image to obtain the user's emotion trend data includes: Perform voice, face or body recognition on the feature image to obtain face and body features; Acquire audio information corresponding to the feature image, extract speech features from the audio information, The facial and body features and voice features are input into the emotion prediction model to obtain the user's emotion trend data.

3. The method according to claim 1, characterized in that The emotional trend data includes first emotional trend data and second emotional trend data; The obtaining of preliminary intention features of the item through the sentiment trend data includes: Determine the first bypass subtree and the final bypass subtree of the decision tree; The first emotion trend data and the second emotion trend data are respectively set as the characteristic attribute values ​​of the first bypass subtree and the final bypass subtree of the decision tree, to obtain the classification result of the output of the decision tree, and the classification result is determined as the preliminary intention feature of the item.

4. The method according to claim 1, characterized in that: The step of performing object recognition on the feature image to obtain initial position information of the object includes: The feature image is input into a deep image recognition model to obtain output information of the initial position of the object.

5. The method according to claim 1, characterized in that The step of combining the preliminary intention feature of the item and the initial location information of the item to calculate the predicted trajectory information of the item includes: If the preliminary intention characteristic of the item is a positive trend, the first probability of the item moving to the first preset location is adjusted; If the preliminary intention characteristic of the item is a negative trend, then adjusting the second probability of the item moving to the second preset location; Generate object prediction trajectory information according to the first probability and the second probability.

6. The method according to claim 2, characterized in that The facial features and voice features are input into the emotion prediction model to obtain the user's emotion trend data, including Combining the facial and body features and voice features into feature training samples; Initializing the learning rate and weight parameters of the emotion prediction model; The aspect ratio between each image pixel in the feature training sample is calculated and normalized using an activation function to obtain the spatial relationship similarity; The feature similarity between each node is calculated by the linearly transformed image pixels in each node; The feature similarity between each speech feature point is calculated by the speech syllable after linear transformation in each speech feature; The specific target similarity between each node contained in the specific target area is obtained by calculating the image pixels of the specific target area; The matrix of node edge weights is constructed through the spatial relationship similarity, specific target similarity and feature similarity between each node in the sentiment prediction model; Construct a fully connected graph of each node consisting of feature graph nodes of feature training samples, edges connecting nodes, and a matrix of node edge weights; The weight parameters are iterated through loss function optimization to obtain a trained emotion prediction model, and then new facial and body features and voice features are input into the trained emotion prediction model to obtain output emotion trend data.

7. An item tracking control system for a smart storage cabinet, characterized in that: include: A first acquisition module is used to acquire a feature image captured by a binocular camera on the smart storage cabinet; An emotion trend data module, used to call a preset emotion prediction model according to the feature image to obtain the user's emotion trend data; A first recognition module, used to obtain preliminary intention features of an item through the sentiment trend data; A second recognition module is used to perform object recognition on the feature image to obtain initial position information of the object; The item prediction trajectory information module is used to combine the item's preliminary intention features and the item's initial position information to calculate the item's prediction trajectory information.

8. The system according to claim 7, characterized in that The emotional trend data module includes A face and body recognition submodule, used for performing voice, face or body recognition on the feature image to obtain face and body features; The audio submodule is used to obtain the audio information corresponding to the feature image and extract the speech features from the audio information. The input submodule is used to input the facial and body features and voice features into the emotion prediction model to obtain the user's emotion trend data.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the item tracking control method of the smart storage cabinet described in any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the item tracking control method of the smart storage cabinet described in any one of claims 1 to 6 are implemented.

Images