Method and system for recognizing and matching goods based on hand movement

Abstract

A hand gesture-based product recognition and product location matching method and system are disclosed, wherein the method tracks the position of an object's hand based on the position of the object's hand in a video recognized by a camera, determines whether there is a product in the object's hand when the object's hand enters and then leaves a product display area where products are located by section, and when it is determined that there is a product in the object's hand, the method identifies the product corresponding to the position where the object's hand left, adds the identified product to a purchase list, and updates the product information of the product display area.

Description

Technology Field

[0001] The present invention relates to a hand gesture-based product recognition and product location matching method and system, and more specifically, to a method and system that performs automatic product selection and payment functions by analyzing user's hand gestures and product location data in real time using a vision camera and an AI analysis program. Background Technology

[0002] Recently, due to rising labor costs, many unmanned stores are opening. An unmanned store is a shop that conducts sales operations solely through shelves and cash registers, without any human staff. However, because it is an unmanned system, it is vulnerable to theft, so it is operated under surveillance via CCTV. Additionally, the system operates by having customers take items from the shelves and scan them individually to make a payment.

[0003] However, the method of payment by scanning products causes inconvenience to buyers, and there are concerns about product theft when buyers make payments without scanning. The problem to be solved

[0004] The problem that the present invention aims to solve is to provide a method and a system that perform automatic product selection and payment functions by analyzing user hand movements and product location data in real time using a vision camera and an AI analysis program to solve the above-mentioned problems. means of solving the problem

[0005] According to one embodiment of the present invention for solving the above technical problem, a hand gesture-based product recognition and product location matching method comprises: a step of tracking the position of an object's hand based on the position of the object's hand in an image recognized by a camera; a step of determining whether there is a product in the object's hand when the object's hand enters and then leaves a product display area where products are located by section; a step of determining a product corresponding to the position where the object's hand left when it is determined that there is a product in the object's hand; and a step of adding the determined product to a purchase list and updating the product information of the product display area.

[0006] A hand gesture-based product recognition and product location matching method preferably further comprises the steps of: determining whether there is a product in the hand of the object when the position of the hand of the object containing the product moves out of the product display area after entering; treating the product corresponding to the position where the object's hand moved out as returned when it is determined that there is no product in the hand of the object; and deleting the returned product from the purchase list and updating the product information of the product display.

[0007] According to another embodiment of the present invention for solving the above technical problem, a hand gesture-based product recognition and product location matching system comprises: a camera that recognizes the hand of an object; a location tracking unit that tracks the position of the object's hand based on the position of the object's hand in an image recognized by the camera; a product determination unit that determines whether there is a product in the object's hand when the object's hand moves out of a product display area where products are located by section after entering the area, and when it is determined that there is a product in the object's hand, specifies a product corresponding to the position where the object's hand moved out; and a control unit that adds the product specified by the product determination unit to a purchase list and updates the product information of the product display area.

[0008] It is preferable that the above product determination unit determines whether there is a product in the object's hand when the position of the object's hand, which has a product, moves out of the product display area after entering it, and if it determines that there is no product in the object's hand, treats the product corresponding to the position where the object's hand moved out as returned, and the control unit deletes the returned product from the purchase list and updates the product information of the product display. Effects of the invention

[0009] According to the present invention, misrecognition can be minimized by matching the location information of a product with the movement pattern of a hand.

[0010] In addition, according to the present invention, when determining whether the hand is empty, it is possible to precisely detect whether the hand possesses a product recognized by the hand by utilizing a product feature point analysis database. Brief explanation of the drawing

[0011] FIG. 1 is a diagram showing a flowchart regarding a hand gesture-based product recognition and product location matching method according to an embodiment of the present invention. FIG. 2 is a flowchart illustrating a deep learning-based hand position recognition and movement tracking method according to an embodiment of the present invention. FIG. 3 is a block diagram illustrating a hand gesture-based product recognition and product location matching system according to an embodiment of the present invention. FIG. 4 is a diagram showing a block diagram of an AI engine according to one embodiment of the present invention. Specific details for implementing the invention

[0012] Preferred embodiments of the present invention will be described in detail with reference to the attached drawings.

[0013] In describing the embodiments of this specification, if it is determined that a detailed description of related known configurations or functions could obscure the gist of this specification, such detailed description is omitted.

[0014] In the present invention, the description of a specific configuration as "including" does not exclude configurations other than said configuration, and means that additional configurations may be included within the scope of the practice of the present invention or the technical concept of the present invention.

[0015] Furthermore, the components shown in the embodiments of the present invention are depicted independently to represent different characteristic functions and do not imply that each component consists of separate hardware or a single software unit. That is, each component is listed and included as a separate component for convenience of explanation; however, at least two of the components may be combined to form a single component, or a single component may be divided into multiple components to perform a function, and such integrated and separated embodiments of each component are included within the scope of the present invention as long as they do not deviate from the essence of the invention.

[0016] Furthermore, some components may not be essential components performing an essential function in the present invention, but may be optional components merely for enhancing performance. The present invention may be implemented by including only the components essential for realizing the essence of the present invention, excluding components used solely for performance enhancement, and a structure including only the essential components, excluding optional components used solely for performance enhancement, is also included within the scope of the rights of the present invention.

[0017] In addition, components expressed as '~part (unit)', 'module', etc. in this specification may consist of two or more components combined into a single component, or a single component may be divided into two or more components according to more detailed functions. Furthermore, each component described below may additionally perform some or all of the functions of other components in addition to the primary function it is responsible for, and it goes without saying that some of the primary functions of each component may be exclusively performed by other components.

[0018] In addition, detailed descriptions of known functions and configurations that may obscure the essence of the invention will be omitted in this specification.

[0019] FIG. 1 is a diagram showing a flowchart regarding a hand gesture-based product recognition and product location matching method according to an embodiment of the present invention.

[0020] Referring to FIG. 1, in step 110, a device that performs a hand gesture-based product recognition and product location matching method (hereinafter referred to as the "product recognition and location matching device") tracks the position of a person's hand based on the position of the person's hand in an image recognized by a camera.

[0021] In one embodiment of the present invention, the camera is preferably a vision camera.

[0022] The product recognition and location matching device receives images from a vision camera in real time, identifies the location of an object—namely, a person's hand—based on the received images, and tracks it.

[0023] In one embodiment of the present invention, the product recognition and location matching device tracks the location and movement of a person's hand in real time through an image analysis AI engine.

[0024] FIG. 2 is a flowchart illustrating a deep learning-based hand position recognition and movement tracking method according to an embodiment of the present invention.

[0025] Referring to FIG. 2, in step 210, a device that performs a deep learning-based hand position recognition and movement tracking method (hereinafter referred to as the 'AI engine') receives image and hand position and movement data recognized through a camera.

[0026] In step 220, the AI ​​engine removes outliers and noise and converts the camera image and hand position and movement data into a learnable form through preprocessing.

[0027] In step 230, the AI ​​engine stores camera images and hand position and movement data and preprocessed camera images and hand position and movement data. The AI ​​engine includes at least one database.

[0028] In step 240, the AI ​​engine uses the input data to train an AI model created according to the purpose.

[0029] The AI ​​model according to the present invention is an AI model for hand position recognition and tracking. The AI ​​model can be generated according to a task based on past camera images and hand position and movement data.

[0030] The AI ​​model according to the present invention may be generated in the AI ​​engine of the present invention, or may be received from an external source and stored in the AI ​​engine.

[0031] In the present invention, the AI ​​engine learns camera images and hand position and movement data to accurately determine the position and movement of the hand. Through learning, the AI ​​engine generates hand position and movement data.

[0032] AI algorithms or deep learning algorithms include Convolutional Neural Networks (CNN), Deep Neural Networks (DNN), Radial Basis Function Neural Networks (RBFN), Recurrent Neural Networks (RNN), Long Short-Term Memory Networks (LSTM), Generative Adversarial Networks (GAN), Autoencoders Deep Networks, Deep Belief Networks, Multilayer Perceptrons (MLP), Self Organizing Maps (SOM), Restricted Boltzmann Machines (RBM), Feedforward Neural Networks (FNN), and Deep Q-Networks (DQN).

[0033] To briefly summarize, Convolutional Neural Networks (CNNs) function as a specific form of artificial intelligence primarily used to understand visual data such as images and videos. Deep Neural Networks (DNNs) are artificial neural networks with multiple hidden layers. Radial Basis Function Networks (RBFNs) are a distinct subset of artificial neural networks that operate by using radial basis functions as activation functions within the hidden layers. Recurrent Neural Networks (RNNs) constitute a specific type of neural network architecture specifically designed to process sequential data by retaining information over time through loops within the network. Long Short-Term Memory Networks (LSTMs) are a sophisticated type of recurrent neural network architecture designed to address the vanished gradient problem of traditional RNNs. Generative Adversarial Networks (GANs) represent a powerful class of machine learning models composed of two interconnected networks called generators and discriminators. Autoencoder deep learning algorithms are a type of deep learning algorithm used for unsupervised learning tasks that operates by compressing input data into latent or compressed representations and then reconstructing the original data as accurately as possible. Deep Belief Networks are complex neural networks composed of multiple layers of probabilistic models. Multilayer Perceptrons (MLPs) are a basic type of artificial neural network composed of multiple layers of interconnected neurons. Self-Organizing Maps (SOMs) are artificial neural networks used for unsupervised learning and pattern recognition. Restricted Boltzmann Machines (RBMs) are a specific type of neural network composed of two layers: a visible layer and a hidden layer. Feedforward Neural Networks (FNNs) are a basic type of artificial neural network in which information flows unidirectionally from the input layer through the hidden layer to the output layer without cycles. Deep Q-Networks (DQNs) are a type of deep reinforcement learning algorithm that combines deep neural networks and Q-learning, enabling machines to learn optimal behavior in complex environments.

[0034] In one embodiment of the present invention, an AI algorithm or deep learning algorithm used for learning may utilize a recurrent neural network (RNN). A recurrent neural network (RNN) is an algorithm that has the advantage of processing sequence data of various lengths and capturing temporal dependencies.

[0035] However, there are no limitations on the AI ​​algorithm or deep learning algorithm used for learning according to the present invention.

[0036] In step 250, the AI ​​engine stores the trained AI model in a designated repository of the AI ​​platform and manages it. Additionally, the AI ​​engine may separately store the training data trained using the AI ​​model in a designated repository and manage it.

[0037] In step 260, the AI ​​engine can accurately track the position of the hand and the hand using the received camera image, the hand position and movement data determined based thereon, and the AI ​​model.

[0038] In step 120, when the position of a person's hand enters and exits a product display area where products are located by section, the product recognition and position matching device determines whether there is a product in the person's hand.

[0039] The product display shelves are divided into sections, and products are arranged by section. Additionally, the product database pre-stores location information for the sections and product information corresponding to each section.

[0040] In step 130, if it is determined that a product is in a person's hand, the product recognition and location matching device identifies a product corresponding to the location where the person's hand has moved away.

[0041] When the location where the human hand has left the device is determined, the product recognition and location matching device extracts section information regarding the location where the human hand has left the device from the product database and obtains product information based on the extracted section information.

[0042] In step 140, the product recognition and location matching device adds the specified product to the purchase list and updates the product information on the product display shelf.

[0043] The product recognition and location matching device detects that a person has taken out a product and adds the corresponding item to the purchase list displayed on the payment device. Additionally, as the product is taken out, the device updates the product database by deleting the product information on the display shelf.

[0044] In step 150, when the position of a person's hand holding a product enters and exits the product display area, the product recognition and position matching device determines whether there is a product in the person's hand.

[0045] In step 160, if it is determined that there is no product in the person's hand, the product recognition and location matching device treats the product corresponding to the location where the person's hand left as having been returned.

[0046] When the location where the human hand has left the device is determined, the product recognition and location matching device extracts section information regarding the location where the human hand has left the device from the product database and obtains product information based on the extracted section information.

[0047] In step 170, the product recognition and location matching device removes the returned product from the purchase list and updates the product information on the product display shelf.

[0048] The product recognition and location matching device determines that a person has put the product back in without purchasing it, and removes the corresponding product from the purchase list displayed on the payment device. Then, as the product is put back in, the product recognition and location matching device updates the product information on the product display shelf stored in the product database.

[0049] In one embodiment of the present invention, the product recognition and location matching device can perform automatic payment based on products registered in the purchase list when a user accesses a payment zone (e.g., an entrance, an automatic payment area, etc.) after finishing shopping.

[0050] FIG. 3 is a block diagram illustrating a hand gesture-based product recognition and product location matching system according to an embodiment of the present invention.

[0051] Referring to FIG. 3, the hand gesture-based product recognition and product location matching system (300) includes a camera (310), a location tracking unit (320), a product judgment unit (330), a control unit (340), and an AI engine (350).

[0052] The position tracking unit (320) tracks the position of a person's hand based on the position of the person's hand in the image recognized by the camera (310).

[0053] In one embodiment of the present invention, the camera (310) is preferably a vision camera.

[0054] The position tracking unit (320) receives images from the vision camera in real time, identifies the position of an object, that is, a person's hand, based on the received images, and tracks it.

[0055] In one embodiment of the present invention, the position tracking unit (320) tracks the position recognition and movement of a person's hand in real time through an AI engine (350).

[0056] FIG. 4 is a diagram showing a block diagram of an AI engine according to one embodiment of the present invention.

[0057] Referring to FIG. 4, the AI ​​engine (350) includes a receiving unit (351), a preprocessing unit (352), an AI model training unit (353), and a tracking unit (354).

[0058] The receiver (351) receives image and hand position and movement data recognized through the camera (310).

[0059] The preprocessing unit (352) removes outliers and noise and converts the camera (310) image and hand position and movement data into a learnable form through preprocessing.

[0060] The camera (310) image and hand position and movement data and the preprocessed camera (310) image and hand position and movement data are stored in a storage unit (not shown). In another embodiment of the present invention, the AI ​​engine may include at least one database.

[0061] The AI ​​model learning unit (353) learns the AI ​​model generated according to the purpose using the input data.

[0062] The AI ​​model according to the present invention is an AI model for recognizing and tracking the position of a hand. The AI ​​model can be generated according to a task based on past camera (310) images and hand position and movement data.

[0063] The AI ​​model according to the present invention may be generated in the AI ​​engine (350) of the present invention, or may be received from an external source and stored in a storage unit.

[0064] In the present invention, the tracking unit (354) learns the camera (310) image and the position and movement data of the hand, and can accurately determine the position and movement of the hand. The AI ​​engine (350) generates the position and movement data of the hand through learning.

[0065] AI algorithms or deep learning algorithms include Convolutional Neural Networks (CNN), Deep Neural Networks (DNN), Radial Basis Function Neural Networks (RBFN), Recurrent Neural Networks (RNN), Long Short-Term Memory Networks (LSTM), Generative Adversarial Networks (GAN), Autoencoders Deep Networks, Deep Belief Networks, Multilayer Perceptrons (MLP), Self Organizing Maps (SOM), Restricted Boltzmann Machines (RBM), Feedforward Neural Networks (FNN), and Deep Q-Networks (DQN).

[0066] To briefly summarize, Convolutional Neural Networks (CNNs) function as a specific form of artificial intelligence primarily used to understand visual data such as images and videos. Deep Neural Networks (DNNs) are artificial neural networks with multiple hidden layers. Radial Basis Function Networks (RBFNs) are a distinct subset of artificial neural networks that operate by using radial basis functions as activation functions within the hidden layers. Recurrent Neural Networks (RNNs) constitute a specific type of neural network architecture specifically designed to process sequential data by retaining information over time through loops within the network. Long Short-Term Memory Networks (LSTMs) are a sophisticated type of recurrent neural network architecture designed to address the vanished gradient problem of traditional RNNs. Generative Adversarial Networks (GANs) represent a powerful class of machine learning models composed of two interconnected networks called generators and discriminators. Autoencoder deep learning algorithms are a type of deep learning algorithm used for unsupervised learning tasks that operates by compressing input data into latent or compressed representations and then reconstructing the original data as accurately as possible. Deep Belief Networks are complex neural networks composed of multiple layers of probabilistic models. Multilayer Perceptrons (MLPs) are a basic type of artificial neural network composed of multiple layers of interconnected neurons. Self-Organizing Maps (SOMs) are artificial neural networks used for unsupervised learning and pattern recognition. Restricted Boltzmann Machines (RBMs) are a specific type of neural network composed of two layers: a visible layer and a hidden layer. Feedforward Neural Networks (FNNs) are a basic type of artificial neural network in which information flows unidirectionally from the input layer through the hidden layer to the output layer without cycles. Deep Q-Networks (DQNs) are a type of deep reinforcement learning algorithm that combines deep neural networks and Q-learning, enabling machines to learn optimal behavior in complex environments.

[0067] In one embodiment of the present invention, an AI algorithm or deep learning algorithm used for learning may utilize a recurrent neural network (RNN). A recurrent neural network (RNN) is an algorithm that has the advantage of processing sequence data of various lengths and capturing temporal dependencies.

[0068] However, there are no limitations on the AI ​​algorithm or deep learning algorithm used for learning according to the present invention.

[0069] The AI ​​engine (350) stores the AI ​​model that has completed training in a storage unit and manages it. In addition, the AI ​​engine (350) may separately store the training data learned using the AI ​​model in a designated storage unit and manage it.

[0070] The tracking unit (354) can accurately track the position of the hand and the hand using the received camera (310) image, the position and movement data of the hand determined based thereon, and an AI model.

[0071] When the position of a person's hand enters and exits the product display area where the products are located by section, the product judgment unit (330) determines whether there is a product in the person's hand.

[0072] The product display stand is divided into sections, and products are arranged by section. Also, the product database (400) has location information of the sections and product information corresponding to the sections stored in advance.

[0073] When it is determined that a product is in a person's hand, the product determination unit (330) identifies a product corresponding to the position where the person's hand has moved away.

[0074] When the location where the person's hand has left the product is determined, the product determination unit (330) extracts section information regarding the location where the person's hand has left the product database and obtains product information according to the extracted section information.

[0075] The control unit (340) adds a specific product to the purchase list and updates the product information of the product display stand.

[0076] The control unit (340) detects that a person has taken out a product and adds the product to the purchase list displayed on the payment device. Then, the product recognition and location matching device updates the product information of the product display stand stored in the product database by deleting it as the product is taken out.

[0077] When the position of a person's hand holding a product enters and exits the product display area, the product judgment unit (330) determines whether there is a product in the person's hand.

[0078] If it is determined that there is no product in a person's hand, the product judgment unit (330) treats the product corresponding to the position where the person's hand left as having been returned.

[0079] When the location where the person's hand has left the product is determined, the product determination unit (330) extracts section information regarding the location where the person's hand has left the product database and obtains product information according to the extracted section information.

[0080] The control unit (340) removes the returned product from the purchase list and updates the product information of the product display stand.

[0081] The control unit (340) determines that a person has put the product back in without purchasing it and deletes the corresponding product from the purchase list displayed on the payment device. Then, the control unit (340) updates the product information of the product display stand stored in the product database (400) by adding it as the product is put back in.

[0082] The payment unit (not shown) can perform automatic payment based on products registered in the purchase list when the user accesses the payment zone (e.g., entrance, automatic payment area, etc.) after finishing shopping. Explanation of the symbols

[0083] Hand gesture-based product recognition and product location matching system: 300 Camera: 310 Location Tracking Section: 320 Product Judgment Unit: 330 Control Unit: 340 AI Engine: 350

Claims

Claim 1 A hand gesture-based product recognition and product location matching method characterized by comprising: a step of tracking the position of an object's hand based on the position of the object's hand in an image recognized by a camera; a step of determining whether there is a product in the object's hand when the object's hand enters and then leaves a product display area where products are located by section; a step of specifying a product corresponding to the position where the object's hand left when it is determined that there is a product in the object's hand; and a step of adding the specified product to a purchase list and updating the product information of the product display area. Claim 2 A hand gesture-based product recognition and product location matching method according to claim 1, characterized by comprising: a step of determining whether there is a product in the hand of the object when the position of the hand of the object with the product moves out of the product display area after entering the product display area; a step of treating the product corresponding to the position where the object's hand moved out as returned when it is determined that there is no product in the hand of the object; and a step of deleting the returned product from the purchase list and updating the product information of the product display area. Claim 3 A hand motion-based product recognition and product location matching system characterized by comprising: a camera that recognizes the hand of an object; a position tracking unit that tracks the position of the object's hand based on the position of the object's hand in an image recognized by the camera; a product determination unit that determines whether there is a product in the object's hand when the object's hand moves out of a product display area where products are located by section after entering, and when it is determined that there is a product in the object's hand, specifies a product corresponding to the position where the object's hand moved out; and a control unit that adds the product specified by the product determination unit to a purchase list and updates product information of the product display area. Claim 4 A hand motion-based product recognition and product location matching system according to claim 3, wherein the product determination unit determines whether there is a product in the object's hand when the position of the object's hand, which has a product, moves out of the product display area after entering it, and when it determines that there is no product in the object's hand, treats the product corresponding to the position where the object's hand moved out as returned, and the control unit deletes the returned product from the purchase list and updates the product information of the product display area.

Images