Goods taking and placing action behavior identification method and system based on grid model

Abstract

The invention relates to the technical field of intelligent identification and monitoring, and provides a commodity taking and placing action behavior identification method and system based on a grid model. The method comprises the following steps: obtaining a customer image through a YOLOX detector; processing the customer image based on an N-tuple loss function through an OSNet reidentification network, and extracting appearance features of the customer image; processing the customer image through an NSA Kalman filtering algorithm, and extracting motion features of the customer image; target similarity is calculated according to the appearance features and the motion features, a cost matrix is obtained, the cost matrix is traversed through a multi-level data association algorithm, the appearance features and the motion features are matched and associated, and customer track recognition is achieved. According to the invention, customer identification and detection precision is improved, and tracking performance and stability are enhanced.

Description

Technical Field

[0001] The present invention relates to the field of intelligent recognition and monitoring technology, and in particular to a method and system for identifying commodity picking and placing actions based on a grid model. Background Art

[0002] In recent years, the rapid development of computer vision, machine learning, sensor technology and artificial intelligence has provided a technical foundation for the realization of unmanned supermarkets. The advancement of algorithms and hardware has made product identification, customer tracking and automatic checkout more feasible and accurate.

[0003] Traditional retailers are facing tremendous pressure from e-commerce and consumers' growing demand for convenient shopping experience, and need to find new ways to attract customers and improve operational efficiency. As an innovative retail model, unmanned supermarkets can provide uninterrupted service, reduce waiting time in queues, reduce labor costs and provide a unique shopping experience, thereby enhancing competitiveness.

[0004] Existing unmanned supermarkets use RFID (radio frequency identification) technology as an important part of the new retail concept, and one of its core technologies is RFID technology. RFID technology is a wireless communication technology that can identify specific targets and read and write related data through radio signals. It is widely used in commodity management and payment settlement in unmanned supermarkets. In unmanned supermarkets, each item is affixed with an RFID electronic tag, which contains detailed information about the item, such as price, name, inventory, etc. When customers are shopping, the RFID system can quickly identify the commodity information through the reader and realize automatic settlement and management of the commodity.

[0005] When customers leave the unmanned supermarket, they pass through the settlement area equipped with an RFID reader / writer. The system will automatically read the RFID tags on the goods, calculate the total amount, and support multiple payment methods, such as mobile payment and bank card payment, to achieve quick checkout. The RFID system in the unmanned supermarket also has an anti-theft function. When unpaid goods pass through the exit, the RFID reader / writer will detect the unchecked goods and trigger the alarm system, thereby effectively preventing the loss of goods.

[0006] Although the use of RFID technology in new retail scenarios such as unmanned supermarkets has brought about improvements in convenience and efficiency, it also has some disadvantages and challenges. This technology has the following shortcomings: Cost issue: The deployment of RFID technology requires the addition of RFID tags to each product, which increases the cost of the product. In addition, merchants need to invest in RFID readers, software systems, and related hardware facilities, which requires a large initial investment.

[0007] Reading accuracy: Although RFID technology has the advantage of multi-target recognition, misreading may occur in actual applications. For example, when multiple tags respond to the reader at the same time, signal interference may occur, resulting in data errors. In addition, the way the label is attached and the material of the product may also affect the reading accuracy. Summary of the invention

[0008] The present invention aims to solve at least one of the technical problems existing in the related art. To this end, the present invention provides a method and system for identifying the behavior of picking and placing goods based on a grid model, so as to improve the recognition and detection accuracy of customers and enhance the tracking performance and stability.

[0009] The present invention provides a method for identifying a commodity picking and placing action based on a grid model, comprising the following steps: S100, collecting a customer's travel video, performing target detection on a current frame image in the travel video using a YOLOX detector, determining a target bounding box, and cropping the current frame image according to the target bounding box to obtain a customer image; S200, processing the customer image through the OSNet re-recognition network based on the N-tuple loss function to extract the appearance features of the customer image; S300, processing the customer image by using the NSA Kalman filter algorithm to extract the motion features of the customer image; S400, calculating the target similarity based on the appearance features and the motion features, obtaining a cost matrix, traversing the cost matrix through a multi-level data association algorithm, matching and associating the appearance features with the motion features, and realizing customer tracking recognition.

[0010] According to a method for identifying a commodity picking and placing action based on a grid model provided by the present invention, the step S100 includes: S110, collecting a video of the customer's movements through a camera; S120, extracting image features of the current frame image in the tracking video through the skeleton network of the YOLOX detector; S130, performing expression capability enhancement processing on the image features through the ECA module of the YOLOX detector to obtain enhanced image features; S140, compressing, extracting and distributing the enhanced image features in sequence through the WAM module of the neck network of the YOLOX detector to determine the target bounding box.

[0011] According to a method for identifying a commodity picking and placing action based on a grid model provided by the present invention, the step S130 includes: The ECA module uses a 1×1 convolution method to achieve cross-channel information interaction of image features, where the value of the convolution kernel is Determined by an adaptive function; The calculation formula of the adaptive function is: in, is the number of channels; is the first empirical parameter; is the second empirical parameter.

[0012] According to a method for identifying a commodity picking and placing action based on a grid model provided by the present invention, the step S140 includes: S141, using the WAM module to compress the channels of the two images processed by the ECA module to , and extract feature information to obtain two compressed images; cascade the two compressed images to obtain one channel number is A combined image of S142, compressing the number of channels of the combined image by a 1×1 convolution method to obtain a combined compressed image with a channel number of 2, and extracting the spatial weights of the two channels in the combined compressed image; S143, assigning spatial weight information to the image processed by the ECA module through a product operation method to determine a target bounding box.

[0013] According to a method for identifying commodity picking and placing actions based on a grid model provided by the present invention, in step S142, the spatial weight calculation formula is: in, The first channel of the combined compressed image The spatial weight at the location; The first channel of the combined compressed image The weight at the position; The second channel of the combined compressed image The spatial weight at the location; The second channel of the combined compressed image The weight at the position; is the first channel of the combined compressed image; The second channel of the combined compressed image.

[0014] According to a method for identifying a commodity picking and placing action based on a grid model provided by the present invention, in the step S200, the N-tuple loss function The calculation formula is: in, For time; is the positive sample value; is the total number of sample categories; For the The number of negative sample groups; is the negative sample value; For classification samples; is the positive sample weight; is the negative sample weight; is the similarity calculation function.

[0015] According to a method for identifying a commodity picking and placing action based on a grid model provided by the present invention, in the step S300, the Kalman filter gain of the NSA Kalman filter algorithm is The calculation formula is: in, is the covariance matrix; is the observation matrix; is the transposed matrix of the observation matrix; To test confidence; is the preset noise matrix.

[0016] The present invention also provides a commodity picking and placing action behavior recognition system based on a grid model, which is used to execute the commodity picking and placing action behavior recognition method based on a grid model as described above, comprising: Cameras to collect videos of customers’ movements; The target detection module is used to detect the target of the current frame image in the tracking video through the YOLOX detector, determine the target bounding box, and crop the current frame image according to the target bounding box to obtain the customer image; The recognition module is used to process customer images through the OSNet recognition network based on the N-tuple loss function to extract the appearance features of the customer images; A detection module, used to process customer images through the NSA Kalman filter algorithm to extract motion features of customer images; The data association module is used to calculate the target similarity based on the appearance features and motion features, obtain the cost matrix, traverse the cost matrix through the multi-level data association algorithm, match and associate the appearance features with the motion features, and realize customer tracking recognition.

[0017] The above one or more technical solutions in the embodiments of the present invention have at least one of the following technical effects: The present invention provides a method and system for identifying commodity picking and placing actions based on a grid model, comprising the following steps: S100, collecting a customer's track video, performing target detection on a current frame image in the track video through a YOLOX detector, determining a target boundary box, and cropping the current frame image according to the target boundary box to obtain a customer image; S200, processing the customer image through an OSNet re-recognition network based on an N-tuple loss function to extract the appearance features of the customer image; S300, processing the customer image through an NSA Kalman filter algorithm to extract the motion features of the customer image; S400, calculating the target similarity based on the appearance features and the motion features to obtain a cost matrix, traversing the cost matrix through a multi-level data association algorithm, matching and associating the appearance features with the motion features, realizing customer track recognition, and then realizing customer multi-target tracking, thereby ensuring tracking accuracy, real-time and stability.

[0018] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 It is a flow chart of a method for identifying a commodity picking and placing action based on a grid model provided by the present invention.

[0021] Figure 2 It is a structural schematic diagram of a commodity picking and placing action behavior recognition system based on a grid model provided by the present invention.

[0022] Reference numerals: 1. Camera; 2. Target detection module; 3. Recognition module; 4. Detection module; 5. Data association module. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical scheme and advantages of the present invention clearer, the technical scheme in the present invention will be clearly and completely described below. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present invention. The following embodiments are used to illustrate the present invention, but cannot be used to limit the scope of the present invention.

[0024] In the description of the embodiments of the present invention, it should be noted that the terms “first”, “second” and “third” are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0025] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiment of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0026] Combine the following Figure 1 The present invention describes a method for identifying a commodity picking and placing action based on a grid model, comprising the following steps: S100, collecting a customer's travel video, performing target detection on a current frame image in the travel video using a YOLOX detector, determining a target bounding box, and cropping the current frame image according to the target bounding box to obtain a customer image; S200, processing the customer image through the OSNet re-recognition network based on the N-tuple loss function to extract the appearance features of the customer image; S300, processing the customer image by using the NSA Kalman filter algorithm to extract the motion features of the customer image; S400, calculating the target similarity based on the appearance features and the motion features, obtaining a cost matrix, traversing the cost matrix through a multi-level data association algorithm, matching and associating the appearance features with the motion features, and realizing customer tracking recognition.

[0027] According to a method for identifying a commodity picking and placing action based on a grid model provided by the present invention, the step S100 includes: S110, collecting a video of the customer's movements through a camera; S120, extracting image features of the current frame image in the tracking video through the skeleton network of the YOLOX detector; S130, performing expression capability enhancement processing on the image features through the ECA module of the YOLOX detector to obtain enhanced image features; S140, compressing, extracting and distributing the enhanced image features in sequence through the WAM module of the neck network of the YOLOX detector to determine the target bounding box.

[0028] According to a method for identifying a commodity picking and placing action based on a grid model provided by the present invention, the step S130 includes: The ECA module uses a 1×1 convolution method to achieve cross-channel information interaction of image features, where the value of the convolution kernel is Determined by an adaptive function; The calculation formula of the adaptive function is: in, is the number of channels; is the first empirical parameter; is the second empirical parameter.

[0029] According to a method for identifying a commodity picking and placing action based on a grid model provided by the present invention, the step S140 includes: S141, using the WAM module to compress the channels of the two images processed by the ECA module to , and extract feature information to obtain two compressed images; cascade the two compressed images to obtain one channel number is A combined image of S142, compressing the number of channels of the combined image by a 1×1 convolution method to obtain a combined compressed image with a channel number of 2, and extracting the spatial weights of the two channels in the combined compressed image; S143, assigning spatial weight information to the image processed by the ECA module through a product operation method to determine a target bounding box.

[0030] According to a method for identifying commodity picking and placing actions based on a grid model provided by the present invention, in step S142, the spatial weight calculation formula is: in, The first channel of the combined compressed image The spatial weight at the location; The first channel of the combined compressed image The weight at the position; The second channel of the combined compressed image The spatial weight at the location; The second channel of the combined compressed image The weight at the position; is the first channel of the combined compressed image; The second channel of the combined compressed image.

[0031] According to a method for identifying a commodity picking and placing action based on a grid model provided by the present invention, in the step S200, the N-tuple loss function The calculation formula is: in, For time; is the positive sample value; is the total number of sample categories; For the The number of negative sample groups; is the negative sample value; For classification samples; is the positive sample weight; is the negative sample weight; is the similarity calculation function.

[0032] According to a method for identifying a commodity picking and placing action based on a grid model provided by the present invention, in the step S300, the Kalman filter gain of the NSA Kalman filter algorithm is The calculation formula is: in, is the covariance matrix; is the observation matrix; is the transposed matrix of the observation matrix; To test confidence; is the preset noise matrix.

[0033] In one embodiment of the present invention, the steps are specifically included: S100, behavior tracking cameras are deployed at all angles in the store to collect videos of customers' movements. The image features of the current frame image in the movement video are extracted through the skeleton network of the YOLOX detector; the image features are enhanced by the ECA module of the YOLOX detector to obtain enhanced image features; the enhanced image features are compressed, extracted and allocated in sequence by the WAM module of the neck network of the YOLOX detector to determine the target bounding box.

[0034] The skeleton network of the YOLOX detector in this embodiment is provided with an ECA module (channel attention module), which can improve the expressiveness of features and make the network pay more attention to the key information of the current task, thereby improving the detection accuracy. The ECA module adopts a local cross-channel interaction method without dimensionality reduction to avoid information loss caused by dimensionality reduction, reduce model complexity, and improve the learning ability of channel attention.

[0035] Specifically, the ECA module only adds a small number of parameters, but can achieve significant performance improvements. The ECA module implements cross-channel information interaction through 1×1 convolution and uses an adaptive function to control the size of the convolution kernel. When the number of channels is large, a large convolution kernel is used, and when the number of channels is small, a small convolution kernel is used. When , the value of the convolution kernel The adaptive function is: in, is the number of channels; is the first empirical parameter; is the second empirical parameter.

[0036] Furthermore, the neck network of the YOLOX detector is provided with a WAM module (weighted adaptive module) to enhance the integration and expression of feature maps during feature extraction. The WAM module dynamically obtains the weights of each feature map by intelligently fusing the input feature maps, and redistributes the weights back to the feature maps, thereby achieving effective association between feature maps. In addition, in order to solve the problem of information loss that may be caused by feature fusion, this embodiment adds an additional branch to each feature layer in the neck network. The original feature layer is directly transferred to the feature layer after feature fusion through the branch without adding additional parameters or significantly increasing the computational burden. By introducing the WAM module and the additional branch, the feature extraction process is optimized, which not only enhances the expressiveness of the features, but also avoids the information loss that may be caused by feature fusion, thereby achieving efficient and accurate feature processing.

[0037] The WAM module omits the size adjustment step and changes the number of intermediate channels when adjusting the channels from a fixed value to a variable value determined by the number of channels of the input feature. Specifically, the WAM module uses a 1×1 convolution method to compress the number of channels of the two images processed by the ECA module to , and extract feature information to obtain two compressed images; cascade the two compressed images to obtain one channel number is combined image.

[0038] The number of channels of the combined image is compressed by 1×1 convolution to obtain a combined compressed image with 2 channels. The spatial weights of the 2 channels in the combined compressed image are extracted. The calculation formula is as follows: in, The first channel of the combined compressed image The spatial weight at the location; The first channel of the combined compressed image The weight at the position; The second channel of the combined compressed image The spatial weight at the location; The second channel of the combined compressed image The weight at the position; is the first channel of the combined compressed image; The second channel of the combined compressed image.

[0039] The spatial weight information is assigned to the image processed by the ECA module through the product operation method, thereby establishing the correlation between the input images, making the final output feature map contain more useful information, improving the ability of the backbone network to extract features, and determining the target bounding box.

[0040] It should be noted that, in this embodiment, two types of WAM modules are provided, namely, an independent convolution type WAM module and a shared convolution type WAM module. Specifically: Using an independent convolution type WAM module, the compression part uses independent 1×1 convolution operations on the two input feature maps to compress the number of channels to the target number of channels and refine the feature information; using a shared convolution type WAM module, the same 1×1 convolution kernel is used in the compression stage to simultaneously perform channel compression and feature refinement on the two input feature maps.

[0041] The main difference between the two types of WAM modules is whether an independent convolution kernel is used for each input feature map. An independent convolution type WAM module may capture more specific feature representations because each convolution kernel can be optimized specifically for one feature map. The shared convolution type WAM module reduces the parameters of the model by using the same convolution kernel, which limits the feature expression capability to a certain extent, but helps to reduce computational complexity and the amount of parameters. This embodiment provides two different types of WAM modules, and the most suitable one can be selected according to different needs and application scenarios. For example, when higher feature expression capabilities are required, an independent convolution type WAM module is selected, and when more efficient calculations are required, a shared convolution type WAM module is selected.

[0042] S200. Process customer images through the OSNet re-identification network based on the N-tuple loss function to extract the appearance features of the customer images. The OSNet re-identification network (full-scale network) is mainly composed of full-scale residual blocks. Each residual block can capture spatial features of a certain scale. Therefore, the network can simultaneously learn the global and local features of the target, and use deep separable convolution instead of ordinary convolution to reduce the number of parameters. A unified aggregation gate is introduced to dynamically fuse the features of different scales extracted by the full-scale residual block into full-scale features based on the weight information. While having the re-identification performance of the deep residual network, the number of parameters is greatly reduced to only 2.2M, which can improve the recognition of appearance features and improve the operating efficiency of the algorithm.

[0043] Specifically, the customer image is used as the input of the OSNet re-recognition network, which includes 1 convolutional layer, 3 residual modules and 1 1×1 convolutional module. Each residual module contains multiple convolutional layers, average pooling layers, batch norm layers and linear rectification functions.

[0044] Inputting a customer image of size 256×128 into the OSNet recognition network can output a feature map of size 16×8.

[0045] In order to reduce the interference between tasks, the OSNet re-identification network sets up three branches, namely the global branch, the feature erasing branch and the local block branch. Among them, the global branch uses the output feature map as the input of the feature pyramid structure to extract the multi-scale feature information of pedestrians; the feature erasing branch is composed of a dual-channel attention feature erasing model, including a channel attention erasing module and a spatial attention erasing module. The channel attention erasing module erases the channels with low attention intensity, and the spatial attention erasing module erases the most significant features according to a certain ratio. The two force the OSNet re-identification network to pay attention to the less significant parts and enhance the receptive field; the local block branch cuts the customer image into several blocks and cascades them into a single feature to extract key local information and improve the accuracy of customer re-identification.

[0046] Furthermore, the N-tuple loss function achieves joint optimization of the distances between multiple samples from multiple categories in the feature space, thereby making the training and testing objectives of the pedestrian re-identification model more consistent. The calculation formula is: in, For time; is the positive sample value; is the total number of sample categories; For the The number of negative sample groups; is the negative sample value; For classification samples; is the positive sample weight; is the negative sample weight; is the similarity calculation function.

[0047] S300, processing the customer image through the NSA Kalman filter algorithm to extract the motion characteristics of the customer image; wherein the NSA Kalman filter algorithm adjusts the observation error based on the detection confidence, and the Kalman filter gain The calculation formula is: in, is the covariance matrix; is the observation matrix; is the transposed matrix of the observation matrix; To test confidence; is the preset noise matrix.

[0048] S400, calculate the target similarity based on the appearance features and motion features, obtain the cost matrix, traverse the cost matrix through the multi-level data association algorithm, match and associate the appearance features with the motion features, and realize customer tracking recognition. Specifically, based on the hierarchical matching method of detection confidence, the tracking items to be matched are classified according to the trajectory state, and different similarity calculation methods are adopted for trajectories in different states. At the same time, in order to improve the matching accuracy, EIOU matching is introduced to replace the original IOU matching.

[0049] Specifically, in this embodiment, the target detection results of the current frame are first divided into two categories based on the detection confidence, where high-confidence detections are preferentially matched by using appearance features to calculate target similarity, and low-confidence detections are matched by calculating the EIOU distance between the detection frame and the Kalman filter prediction frame, and detections with confidence greater than 0.6 are classified as high-confidence detections. When performing appearance feature matching, the tracking trajectories are divided into two categories, namely, non-lost trajectories in the activated state and lost trajectories in the inactivated state. Trajectories that have not been re-matched for 30 consecutive frames will be marked as lost.

[0050] For the trajectory of the activated state, in order to avoid the interference of high similarity targets, the cosine distance is combined with the Mahalanobis distance, the displacement of the target is restricted to a certain extent, and a joint metric is constructed to detect and track the distance. The calculation formula is: in: For the Tracking trajectories; For the A detection box; It is a hyperparameter that controls the influence of different distances on the cost matrix; in this embodiment, the value is 0.98; For the The feature vector of the detection box is The cosine distance between the most recently updated feature vectors of the tracking trajectories; For the Tracking trajectories to The Mahalanobis distance of the detection boxes.

[0051] in, The calculation of is based on the exponential moving average method, and the calculation formula is: in: is the exponential factor; is the index of the current frame; For the The detected appearance features matched by the tracking trajectories; is the momentum term, and its value is 0.9 in this embodiment.

[0052] For inactive trajectories, only the cosine distance is used to calculate the appearance similarity, but no longer a single feature vector is used. Instead, the detected feature vector is calculated with the nearest distance retained in the tracking trajectory. (In this embodiment The value of is 30) the average value of the cosine distance of the detected feature vectors , the calculation formula is: in: For the The feature vector of the detection box; For the The tracking trajectories are retained The cosine distance of the eigenvectors.

[0053] For the detection frame With prediction box , the distance of IOU matching , the calculation formula is: when and The IOU distance cannot reflect the relationship between the two. and The position situation between them, that is, the processing ability of occluded targets and lost target recovery is limited.

[0054] The EIOU distance It is an advanced version of the IOU distance, and its calculation formula is: in: is the width of the minimum bounding box of the detection box and the prediction box; is the height of the minimum bounding box of the detection box and the prediction box; is the coordinate of the center point of the detection box; is the coordinate of the center point of the prediction box; is the width of the detection box; is the width of the prediction box; is the height of the detection frame; is the height of the prediction box; is the Euclidean distance Using EIOU distance to replace IOU distance can better judge the position between the detection box and the prediction box and improve the matching accuracy.

[0055] The following is a description of the commodity picking and placing action behavior recognition system based on a grid model provided by the present invention. The commodity picking and placing action behavior recognition system based on a grid model described below and the commodity picking and placing action behavior recognition method based on a grid model described above can be referred to each other.

[0056] like Figure 2 As shown, the present invention also provides a commodity picking and placing action behavior recognition system based on a grid model, which is used to execute the commodity picking and placing action behavior recognition method based on a grid model as described above, comprising: Camera 1, used to collect videos of customers' movements; The target detection module 2 is used to perform target detection on the current frame image in the tracking video through the YOLOX detector, determine the target bounding box, and crop the current frame image according to the target bounding box to obtain the customer image; Recognition module 3, used to process customer images through OSNet recognition network based on N-tuple loss function to extract appearance features of customer images; Detection module 4, used to process customer images through NSA Kalman filter algorithm to extract motion features of customer images; The data association module 5 is used to calculate the target similarity based on the appearance features and the motion features, obtain the cost matrix, traverse the cost matrix through the multi-level data association algorithm, match and associate the appearance features with the motion features, and realize customer tracking recognition.

[0057] The application scenario of the commodity picking and placing action recognition system based on the grid model provided by the present invention is: By registering with the customer mini program, we collect relevant customer information, including name, mobile phone number and online payment, and then generate an ID bound to the customer. The customer enters the store by scanning facial recognition or the mini program code and authorizes password-free payment. The system obtains the customer's ID, which facilitates the tracking of subsequent customer shopping behavior.

[0058] Behavior tracking cameras are deployed at various angles in the store to collect customer behavior trajectories. When a customer enters the identification area from the entrance, the system begins to track his or her behavior trajectory. In the store, the system divides the identification area into grid areas one by one. The width of the grid area is adapted to the size of the gravity shelf and is encoded. Each layer of the gravity shelf is composed of multiple transparent grids, and the width of each grid is the same as the width of the grid area.

[0059] When a customer takes a product from the shelf, the gravity sensor of the gravity shelf transmits the gravity change data back to the system. The product data of each layer on the gravity shelf is recorded in the system. According to the change in gravity and the location where the change occurs, the system can retrieve the name, price and quantity of the product taken. And put the product into the virtual shopping cart corresponding to the customer's ID.

[0060] The customer takes the merchandise and goes directly out of the store. The camera monitors the customer leaving the store and the system starts settling the customer's bill. The system calls up the customer's virtual shopping cart based on the customer ID provided by the identification method and calculates the customer's bill price based on the types and quantities of merchandise in the cart. As the customer has authorized this password-free payment, the system pushes the bill to the payment software, which deducts the corresponding amount based on the bill amount, thus ending the purchase process.

[0061] The above one or more technical solutions in the embodiments of the present invention have at least one of the following technical effects: The present invention provides a method and system for identifying commodity picking and placing actions based on a grid model, comprising the following steps: S100, collecting a customer's track video, performing target detection on a current frame image in the track video through a YOLOX detector, determining a target boundary box, and cropping the current frame image according to the target boundary box to obtain a customer image; S200, processing the customer image through an OSNet re-recognition network based on an N-tuple loss function to extract the appearance features of the customer image; S300, processing the customer image through an NSA Kalman filter algorithm to extract the motion features of the customer image; S400, calculating the target similarity based on the appearance features and the motion features to obtain a cost matrix, traversing the cost matrix through a multi-level data association algorithm, matching and associating the appearance features with the motion features, realizing customer track recognition, and then realizing customer multi-target tracking, thereby ensuring tracking accuracy, real-time and stability.

[0062] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for identifying commodity picking and placing actions based on a grid model, characterized in that: The steps include: S100, collecting a customer's travel video, performing target detection on a current frame image in the travel video using a YOLOX detector, determining a target bounding box, and cropping the current frame image according to the target bounding box to obtain a customer image; S200, processing the customer image through the OSNet re-recognition network based on the N-tuple loss function to extract the appearance features of the customer image; S300, processing the customer image by using the NSA Kalman filter algorithm to extract the motion features of the customer image; S400, calculating the target similarity based on the appearance features and the motion features, obtaining a cost matrix, traversing the cost matrix through a multi-level data association algorithm, matching and associating the appearance features with the motion features, and realizing customer tracking recognition.

2. The method for identifying commodity picking and placing actions based on a grid model according to claim 1, characterized in that: The step S100 includes: S110, collecting a video of the customer's movements through a camera; S120, extracting image features of the current frame image in the tracking video through the skeleton network of the YOLOX detector; S130, performing expression capability enhancement processing on the image features through the ECA module of the YOLOX detector to obtain enhanced image features; S140, compressing, extracting and distributing the enhanced image features in sequence through the WAM module of the neck network of the YOLOX detector to determine the target bounding box.

3. The method for identifying commodity picking and placing actions based on a grid model according to claim 2, characterized in that: The step S130 includes: The ECA module uses a 1×1 convolution method to achieve cross-channel information interaction of image features, where the value of the convolution kernel is Determined by an adaptive function; The calculation formula of the adaptive function is: in, is the number of channels; is the first empirical parameter; is the second empirical parameter.

4. The method for identifying commodity picking and placing actions based on a grid model according to claim 2, characterized in that: The step S140 includes: S141, using the WAM module to compress the channels of the two images processed by the ECA module to , and extract feature information to obtain two compressed images; cascade the two compressed images to obtain one channel number is A combined image of S142, compressing the number of channels of the combined image by a 1×1 convolution method to obtain a combined compressed image with a channel number of 2, and extracting the spatial weights of the two channels in the combined compressed image; S143, assigning spatial weight information to the image processed by the ECA module through a product operation method to determine a target bounding box.

5. The method for identifying commodity picking and placing actions based on a grid model according to claim 4, characterized in that: In the step S142, the spatial weight calculation formula is: in, The first channel of the combined compressed image The spatial weight at the location; The first channel of the combined compressed image The weight at the position; The second channel of the combined compressed image The spatial weight at the location; The second channel of the combined compressed image The weight at the position; is the first channel of the combined compressed image; The second channel of the combined compressed image.

6. The method for identifying commodity picking and placing actions based on a grid model according to claim 1, characterized in that: In the step S200, the N-tuple loss function The calculation formula is: in, For time; is the positive sample value; is the total number of sample categories; For the The number of negative sample groups; is the negative sample value; For classification samples; is the positive sample weight; is the negative sample weight; is the similarity calculation function.

7. The method for identifying commodity picking and placing actions based on a grid model according to claim 1, characterized in that: In the step S300, the Kalman filter gain of the NSA Kalman filter algorithm The calculation formula is: in, is the covariance matrix; is the observation matrix; is the transposed matrix of the observation matrix; To test confidence; is the preset noise matrix.

8. A commodity picking and placing action recognition system based on a grid model, characterized in that: The method for identifying a commodity picking and placing action based on a grid model according to any one of claims 1 to 7 comprises: Cameras to collect videos of customers’ movements; The target detection module is used to detect the target of the current frame image in the tracking video through the YOLOX detector, determine the target bounding box, and crop the current frame image according to the target bounding box to obtain the customer image; The recognition module is used to process customer images through the OSNet recognition network based on the N-tuple loss function to extract the appearance features of the customer images; A detection module, used to process customer images through the NSA Kalman filter algorithm to extract motion features of customer images; The data association module is used to calculate the target similarity based on the appearance features and motion features, obtain the cost matrix, traverse the cost matrix through the multi-level data association algorithm, match and associate the appearance features with the motion features, and realize customer tracking recognition.

Images