Commodity identification method, device and equipment for unmanned container and medium

Abstract

The invention relates to a commodity identification method, device and equipment for an unmanned container and a medium, and belongs to the technical field of deep learning, and the commodity identification method for the unmanned container comprises the following steps: collecting video data of commodity taking of a user through a multi-view camera of the unmanned container; each frame of image data in the video data is detected based on a commodity image detection network to obtain a commodity image, the commodity image is identified based on a constructed MDFM network to obtain a commodity identification result, and the MDFM network comprises a multi-scale feature fusion module, a parallel Transform module, a Mama module, a frequency domain processing module and a detection module. And the commodity identification accuracy is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning, and in particular to a commodity identification method and device for unmanned cabinets, equipment and medium. BACKGROUND

[0002] The unmanned vending mode can complete the sale of goods without relying on manual operation, which not only reduces the operating cost and improves the settlement speed, but also optimizes the shopping experience of consumers. Among them, the commodity identification technology is the core technology of the unmanned retail shopping experience.

[0003] In the scenario of an unmanned cabinet, the commodity identification system needs to accurately identify the type, quantity and spatial position of the commodity in a large amount of image data, which not only requires the system to have efficient image processing capability, but also needs to have certain intelligent decision-making capability to cope with the uncertainty of complex environment and consumer behavior. In the actual application scenario, there are problems such as image blur, deformation and partial occlusion when the unmanned cabinet camera dynamically captures commodity pictures, which leads to a significant decline in recognition performance.

[0004] With the rise of deep learning, convolutional neural networks have achieved great success in image processing. Various large CNN networks have been proven to be able to directly undertake feature extraction work for various image processing tasks after pre-training. However, CNN networks have limitations due to the influence of the receptive field. Vision Transformer (ViT) has become a research hotspot for image recognition and other tasks due to its advantages in capturing complex features and global context information. Traditional ViT divides images into fixed-size patches for feature extraction, which has poor adaptability to input size and insufficient use of global information, thereby leading to low accuracy of commodity identification. SUMMARY

[0005] Therefore, it is necessary to provide a commodity identification method, device, equipment and medium for an unmanned cabinet to solve the technical problem of low accuracy of commodity identification in the scenario of an unmanned cabinet.

[0006] In order to solve the above problems, in a first aspect, the present application provides a commodity identification method for an unmanned cabinet, comprising: acquiring video data of a user taking goods by a multi-view camera of an unmanned cabinet; detecting each frame of image data in the video data based on a commodity image detection network to obtain a commodity image; The MDFM network is constructed to recognize the product image and obtain a product recognition result, wherein the MDFM network comprises a multi-scale feature fusion module, a parallel Transformer module, a Mamba module, a frequency domain processing module, and a detection module; the Transformer module is configured to extract global features of the product image; the Mamba module is configured to extract context modeling features of the product image; the frequency domain processing module is configured to extract edge detail features of the product image; the multi-scale feature fusion module is configured to fuse the global features, the context modeling features, and the edge detail features to obtain combined features; and the detection module is configured to detect the combined features to obtain the product recognition result.

[0007] In a possible implementation, the product image detection network comprises a backbone network, a neck network, and a head network, and the product image detection network is used to detect each frame of image data in the video data to obtain a product image, comprising: The backbone network is used to extract multi-scale features of each frame of image data in the video data to obtain first image features, second image features, third image features, and fourth image features; The neck feature pyramid of the neck network is used to fuse the first image features, the second image features, the third image features, and the fourth image features to obtain three-scale features; The classification branch of the head network is used to classify the three-scale features to obtain a product image, a non-product image, and a product category, and the regression branch of the head network is used to correct a target boundary of the product image to determine a position of the product, wherein the non-product image comprises a face image.

[0008] In a possible implementation, the backbone network comprises a first basic residual block, a second basic residual block, a first bottleneck residual block, and a second bottleneck residual block; and the backbone network is used to extract multi-scale features of each frame of image data in the video data to obtain first image features, second image features, third image features, and fourth image features, comprising: The first basic residual block and the second basic residual block are respectively used to extract shallow features of each frame of image data to obtain the first image features and the second image features, wherein the first basic residual block and the second basic residual block have the same structure, and the first basic residual block comprises a convolution layer, a batch normalization layer, an activation function, and a residual connection; The first bottleneck residual block and the second bottleneck residual block are respectively used to extract deep features of each frame of image data to obtain the third image features and the fourth image features.

[0009] In a possible implementation, the multi-scale feature fusion module comprises a first multi-head attention module and a second multi-head attention module; the MDFM network is used to identify the product image and obtain a product identification result, comprising: inputting the product image into the MDFM network, performing feature extraction on the product image through a shared convolution of the MDFM network, and obtaining a bottom layer feature; respectively inputting the bottom layer feature into the Transformer module, the Mamba module and the frequency domain processing module, respectively obtaining a global feature, a context modeling feature and an edge detail feature, and respectively performing convolution projection on the global feature, the context modeling feature and the edge detail feature to obtain a first feature vector sequence, a second feature vector sequence and a third feature vector sequence; performing spatial alignment on the first feature vector sequence, the second feature vector sequence and the third feature vector sequence based on the first multi-head attention module to obtain a global spatial feature; performing channel dimension splicing on the first feature vector sequence, the second feature vector sequence and the third feature vector sequence based on the second multi-head attention module to obtain a global channel feature; performing weighted fusion on the global spatial feature and the global channel feature by using a gating fusion mechanism to obtain a merged feature; detecting the merged feature based on the detection module to obtain a product identification result.

[0010] In a possible implementation, the Transformer module comprises a block embedding layer, a depth separable convolution layer, a normalization layer, an activation function, a first ViT module, a second ViT module, a third ViT module and a convolution layer.

[0011] In a possible implementation, the Mamba module comprises a first VSS module, a second VSS module, a third VSS module and a convolution layer; the frequency domain processing module comprises a Fourier transform layer, a separation layer, a frequency domain filtering layer, a frequency domain network module, an inverse Fourier transform layer, a convolution layer and a residual connection; and the detection module comprises a convolution layer, a normalization layer and an activation layer.

[0012] In a possible implementation, the loss function of the MDFM network is: , wherein, is a loss function, is a benchmark margin parameter, is a global and local adjustment coefficient, is a feature level depth parameter, a frequency domain feature weight adjustment coefficient, a frequency domain and spatial domain adjustment coefficient, a first a category of the commodity, a product feature vector and a weight vector between the target product category, a dynamic marginal parameter, a product category, a category label of the product.

[0013] In a second aspect, the present application further provides a product identification device of an unmanned cabinet, comprising: a data acquisition module, configured to acquire video data of a user taking goods through a multi-view camera of the unmanned cabinet; a product image acquisition module, configured to detect each frame of image data in the video data based on a product image detection network to obtain a product image; a product identification module, configured to identify the product image based on a constructed MDFM network to obtain a product identification result, wherein the MDFM network comprises a multi-scale feature fusion module, a parallel Transformer module, a Mamba module, a frequency domain processing module, and a detection module, the Transformer module is configured to extract global features of the product image, the Mamba module is configured to extract context modeling features of the product image, the frequency domain processing module is configured to extract edge detail features of the product image, the multi-scale feature fusion module is configured to fuse the global features, the context modeling features, and the edge detail features to obtain merged features, and the detection module is configured to detect the merged features to obtain the product identification result.

[0014] In a third aspect, the present application further provides a product identification device, comprising a processor and a memory; the memory stores a computer readable program which can be executed by the processor; the processor executes the computer readable program to realize the steps in the product identification method of the unmanned cabinet.

[0015] In a fourth aspect, the present application further provides a computer readable storage medium for storing computer readable programs or instructions, which can realize the steps in the product identification method of the unmanned cabinet when executed by a processor.

[0016] The application has the beneficial effects that: video data of user taking goods is collected by the multi-view camera of the unmanned cabinet, each frame of image data in the video data is detected based on a commodity image detection network to obtain a commodity image, accurate positioning and preliminary classification of the commodity in the unmanned cabinet scene are realized, a reliable detection basis is provided for subsequent commodity identification, the commodity image is identified based on the constructed MDFM network to obtain a commodity identification result, wherein the MDFM network includes a multi-scale feature fusion module, a parallel Transformer module, a Mamba module, a frequency domain processing module and a detection module, through the parallel Transformer module, the Mamba module and the frequency domain processing module in the MDFM network, the image features have higher global semantic abstractness, the low-frequency information strengthens the overall distribution, and the high-frequency information has strong representation on the edge details, cross-angle and multi-domain feature extraction is realized, feature interaction is realized through the multi-scale feature fusion module of the MDFM network, and the accuracy of commodity identification is improved. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0018] Figure 1 An embodiment flow chart of the commodity identification method of the unmanned cabinet provided by the present application; Figure 2 A structure schematic diagram of the MDFM network of the commodity identification method of the unmanned cabinet provided by the present application; Figure 3 A structure schematic diagram of an embodiment of the commodity identification device of the unmanned cabinet provided by the present application; Figure 4 A structure schematic diagram of an embodiment of the commodity identification device provided by the present application. DETAILED DESCRIPTION

[0019] The preferred embodiments of the present application will be specifically described below in combination with the drawings, wherein the drawings constitute a part of the present application, and are used to illustrate the principles of the embodiments of the present application, and are not used to limit the scope of the present application.

[0020] Reference to an "embodiment" in this document means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are they necessarily mutually exclusive of one another. As will be apparent to those of ordinary skill in the art, embodiments described herein can be combined with other embodiments.

[0021] The application discloses a commodity identification method, device, equipment and medium of unmanned goods cabinet, which can be used in a computer. The method, device or computer readable storage medium involved in the application can be integrated with the above-mentioned device, or can be relatively independent.

[0022] One specific embodiment of the application discloses a commodity identification method of unmanned goods cabinet, which can be executed by a computer, and specifically can be executed by one or more processors of the computer. As shown in the figure, the commodity identification method of unmanned goods cabinet comprises the following steps. Figure 1 S101, collecting video data of a user taking goods through a multi-view camera of an unmanned goods cabinet.

[0023] S102, detecting each frame of image data in the video data based on a commodity image detection network to obtain a commodity image. It should be noted that the commodity image detection network is a LWLS (Lightweight and with limited resolution, LWLS) network, the LWLS network comprises a backbone network, a neck network and a head network, through shallow feature extraction and deep feature extraction of the backbone network, complementary of detail features and semantic features is realized, the feature misplacement problem caused by angle change in multi-view commodity detection is effectively alleviated, and the feature representation ability for stacked and shielded commodities is enhanced.

[0024] S103, identifying the commodity image based on a constructed MDFM network to obtain a commodity identification result, wherein the MDFM network comprises a multi-scale feature fusion module, a parallel Transformer module, a Mamba module, a frequency domain processing module and a detection module, the Transformer module is used for extracting global features of the commodity image, the Mamba module is used for extracting context modeling features of the commodity image, the frequency domain processing module is used for extracting edge detail features of the commodity image, the multi-scale feature fusion module is used for fusing the global features, the context modeling features and the edge detail features to obtain combined features, and the detection module is used for detecting the combined features to obtain the commodity identification result. It should be noted that the identification of the commodity image through the MDFM network realizes feature extraction across multiple angles and multiple domains, and improves the accuracy of image recognition. ​

[0025] In some embodiments, in step S101, video data of the user picking up goods is collected by the multi-view camera of the unmanned vending machine. The multi-view camera deployed in the vending machine is used for real-time monitoring and collection, and the entire video of the user picking up goods is extracted. The video is decomposed into images frame by frame in offline mode to obtain image data.

[0026] In some embodiments, in step S102, each frame of image data in the video data is detected based on the product image detection network to obtain a product image. The product image detection network includes a backbone network, a neck network, and a head network. Multi-scale feature extraction is performed on each frame of image data in the video data based on the backbone network to obtain a first image feature, a second image feature, a third image feature, and a fourth image feature. The backbone network includes a first basic residual block, a second basic residual block, a first bottleneck residual block, and a second bottleneck residual block. Shallow feature extraction is performed on each frame of image data based on the first basic residual block and the second basic residual block to obtain the first image feature and the second image feature. The first basic residual block and the second basic residual block have the same structure. The first basic residual block includes a convolutional layer, a batch normalization layer, an activation function, and a residual connection. Deep feature extraction is performed on each frame of image data based on the first bottleneck residual block and the second bottleneck residual block to obtain the third image feature and the fourth image feature. For the input image, in the scenario of recognizing goods in unmanned vending machines across multiple angles, considering complex situations such as product tilting, stacking, occlusion, and similar packaging, and adapting to the computing power constraints of edge computing devices, multi-scale features are first extracted through a backbone network optimized by computational redistribution. The number of channels, feature map resolution, and computing resources are dynamically adjusted at different stages based on the target computational load. The backbone network search space comprises four stages, each using different types of residual blocks according to feature extraction requirements. The first two stages are shallow feature extraction stages, both employing basic residual blocks to focus on capturing details such as product packaging texture and local contours. Each basic residual block consists of two... Convolutional layers are concatenated, with each convolutional layer followed by a batch normalization layer and a ReLU activation function. Residual connections are established through identity mapping or... The convolutional matching of channel numbers, followed by element-wise feature fusion, enhances the basic detection capability for low-resolution product targets and avoids loss of detailed information. The latter two stages are deep feature extraction stages, both employing bottleneck residual blocks. These blocks encode global semantic information of the product while controlling parameter redundancy. The bottleneck residual blocks are composed of… Convolutional layers Convolutional layers The convolutional layers are connected in series, where, Convolutional layers are responsible for channel compression and restoration. Convolutional layers are responsible for semantic feature extraction, and the entire network stabilizes gradient propagation through residual connections, reducing computational cost while maintaining semantic expressiveness and adapting to the computing power constraints of edge devices. Simultaneously, the backbone network configures the stacking quantity of each residual block at each stage. Number of channels per residual block Two parameters, by dynamically adjusting the number of stacked residual blocks and the number of channels in the residual blocks, enable precise allocation of computing resources at different feature extraction stages. For example, in the shallow feature extraction stage, a larger parameter can be set. To retain more detailed feature channels, the depth stage can be appropriately reduced. To reduce redundant computation, the number of channels in the shallow stage is set to 64-128 to retain more feature channels, and the feature map resolution is maintained at [specific value]. To avoid losing details in partially occluded areas of the product, 30% of computing resources are allocated to ensure the capture of local details such as product packaging textures and text labels; the number of channels in the deeper stages is gradually increased to 256-512 to encode semantic features of product categories, through... Max pooling reduces the feature map resolution to To focus on the overall shape of the product, only 30% of computing resources are allocated to avoid redundant computation. In the shallow stage, the backbone network uses basic residual blocks to enhance the target detection capability and better cope with problems such as detail deformation caused by multi-angle shooting and texture blurring caused by lighting changes. In the deep stage, bottleneck residual blocks or depthwise separable convolutions are used to reduce parameter redundancy, realize efficient inference of the model on edge devices, and balance feature extraction accuracy and computational efficiency. The neck feature pyramid based on the neck network fuses the first, second, third, and fourth image features to obtain three-scale features. These multi-scale features are then fed into a lightweight neck network. The neck network fuses semantic information from different levels through the feature pyramid. The neck feature pyramid, through top-down feature propagation and lateral connections, fuses the multi-scale features output from the four stages of the backbone network to output three-scale features, including large-scale, medium-scale, and low-scale features. For example, the semantic features from the deep stages of the backbone network can be upsampled to improve resolution and then added element-wise or concatenated with the detailed features from the shallow stages to achieve complementarity between detailed and semantic features. This effectively alleviates the feature misalignment problem caused by angle changes in multi-view product detection and enhances the feature representation ability for stacked and occluded products. The classification branch of the head network classifies the three-scale features to obtain product images, non-product images, and product categories. The regression branch of the head network then refines the target boundaries of the product images to determine the product's location. Non-product images include face images. The head network is constructed based on shared multi-scale features and stacked convolutional layers. It shares the three-scale features processed by the neck feature pyramid. The search space is defined by two parameters: the number of blocks *m* and the number of channels *n*. Structurally, the head network consists of *m* stacked convolutional layers, each with *h* channels. Each convolutional layer is followed by a batch normalization layer and a ReLU activation function to enhance non-linear expression. The stacked convolutional layers further refine the shared three-scale features before processing them through classification and regression branches. The classification branch... The convolutional layer outputs a classification score for the product category, achieving a preliminary distinction between products and non-products, and between different product categories; the regression branch also... The convolutional layer outputs bounding box offsets to accurately correct the boundaries of detected product targets, ultimately achieving accurate product positioning and preliminary category identification in unmanned vending machine scenarios, providing a reliable detection foundation for subsequent product recognition.

[0027] After detecting face images and product images, a visual anonymization method combining reversible pixel obfuscation and asymmetric encryption is used to encrypt the face images. The mosaic pattern generated by this method is visually unrecognizable and, due to its encryption properties, effectively prevents data from being maliciously restored, providing cryptographic-level security for customer privacy. On the other hand, product images are annotated with information such as product type, location, and bounding box. The product image area within the bounding box is then cropped to generate a product image dataset. Ultimately, this achieves intelligent processing with "one image, two uses": while strongly protecting privacy, it accurately annotates and crops product areas, automatically generating a high-quality product image dataset.

[0028] After obtaining the product image dataset, it can be augmented. Besides geometric transformations, color changes, and noise additions, the distribution, categories, and missing elements of the product image data can be analyzed, such as poor lighting conditions (highlights, high occlusion), and objects at specific angles. A multi-view image generation model based on a diffusion model can then be designed to generate new images. The process involves using a basic diffusion model to generate the main image portion, while simultaneously designing a camera and geometric condition guide to provide different types of guidance from the acquired images to the image model. A multi-view attention layer and an image cross-attention layer are introduced to avoid invasive modifications to the basic model. Secondly, the multi-view attention layer and the image cross-attention layer are combined with the original diffusion model to form a dual-branch structure, ensuring that the attention layer shares input features with the pre-trained self-attention layer, fully inheriting the original prior information. Finally, based on different application requirements, multi-view attention strategies (diagonal self-attention and Z-shaped self-attention) are designed to adapt to different multi-view generation needs. During training, the model learns from a large-scale existing public dataset to acquire features of prior knowledge of the 3D structure of objects. During inference, based on a single input view, the 3D structure of the object is inferred, and multi-view images are generated from different angles to augment the commodity dataset.

[0029] In some embodiments, in step S103, the product image is recognized based on the constructed MDFM network to obtain a product recognition result. The MDFM network includes a multi-scale feature fusion module, a parallel Transformer module, a Mamba module, a frequency domain processing module, and a detection module. The Transformer module extracts global features from the product image, the Mamba module extracts contextual modeling features, the frequency domain processing module extracts edge detail features, the multi-scale feature fusion module fuses global features, contextual modeling features, and edge detail features to obtain merged features, and the detection module detects the merged features to obtain the product recognition result. For a schematic diagram of the MDFM network structure, please refer to [link to schematic diagram]. Figure 2 ,like Figure 2As shown, MDFM (Multi-Domain Fusion Model) includes a multi-scale feature fusion module (Fusion Model fusion module), a Transformer module (Transformer branch), a Mamba module (Mamba branch), and a frequency domain processing module (FFT branch). The Transformer module includes patch embedding layers, depthwise separable convolutional layers, normalization layers (Norm), activation layers (GELU activation layers), a first ViT module (Mobile ViT module 1), a second ViT module (Mobile ViT module 2), a third ViT module (Mobile ViT module 3), and convolutional layers (…). Conv); The Mamba module includes a first VSS module (VSS module 1), a second VSS module (VSS module 2), a third VSS module (VSS module 3), and a convolutional layer ( Conv); The frequency domain processing module includes a Fourier transform layer (FFT2), a splitting layer (Split real / imag real number separation), and a frequency domain filtering layer (Corp low-frequency center). ), Frequency domain network module (FreqNet module), Inverse Fourier Transform layer (IFFT2), Convolutional layer ( Conv) and residual connections ( Conv); The multi-scale feature fusion module includes a first multi-head attention module (attention module_A) and a second multi-head attention module (attention module_B), and the detection module includes convolutional layers ( Conv, normalized layer (Layer) The network consists of a Normalized Convolutional Layer (Norm) and an Activation Layer (ReLU Activation Layer). The product image is input into the MDFM network, where shared convolutions (Stem convolutions) extract features from the product image to obtain low-level features. These low-level features are then input into the Transformer module, Mamba module, and frequency domain processing module to obtain global features, context modeling features, and edge detail features, respectively. Convolutional projections are then performed on these global features, context modeling features, and edge detail features to obtain a first feature vector sequence, a second feature vector sequence, and a third feature vector sequence. Based on the first multi-head attention module, these three feature vector sequences are spatially aligned to obtain global spatial features. Based on the second multi-head attention module, these three feature vector sequences are concatenated along the channel dimension to obtain global channel features. A gated fusion mechanism is used to weightedly fuse the global spatial features and global channel features to obtain merged features. These merged features are then detected by the detection module, which performs convolution, normalization, and activation processing on them to obtain the product recognition result. In short, a single image from the product image is input into the MDFM network with an input size of [size missing]. The RGB image is processed by extracting the low-level feature F0 through a shared convolutional Stem. F0 is then input in parallel into three branches (Transformer module, Mamba module, and frequency domain processing module). The Transformer module branch outputs the global feature. ,like Mamba module branch output context modeling features Frequency domain processing module branch output edge detail features Global features, contextual modeling features, and edge detail features are all To facilitate multi-branch interaction, the format applies different methods to global features, contextual modeling features, and edge detail features. Convolutional projection, with a unified number of channels D, flattens the three features along the spatial dimension into a sequence of feature vectors. ,in, , Then, based on the feature vector sequence, global features are constructed and fusion is completed, thereby realizing feature extraction across multiple angles and domains, which meets the core requirement of dual-path multi-domain integration of multi-source features. The three feature vector sequences are used to construct global features from two dimensions. Global spatial features are obtained by concatenating them in the spatial dimension, and global channel features are obtained by concatenating them in the channel dimension. The multi-scale feature fusion module is based on two independent multi-head attention modules, namely Attention Module A and Attention Module B. These modules, combined with feature reconstruction and gating fusion mechanisms, form a complete module. The multi-scale feature fusion module constructs global features from the three feature vector sequences from two dimensions, and then concatenates them in the spatial dimension through the first multi-head attention module to obtain global spatial features. Global channel features are obtained by concatenating them along the channel dimension using a second multi-head attention module. Two independent multi-head attention modules are introduced to process two types of global features respectively. The former is used to model the spatial correlation between multi-branch feature vector sequences, realizing cross-branch spatial alignment and compensation, while the latter is used to integrate multi-source information in the channel dimension, realizing frequency band and channel complementarity and redistribution. After the interaction is completed, the updated... and The two paths are reconstructed into spatial feature formats, and then a gated fusion mechanism is used. A learnable attention module generates spatially adaptive and channel-adaptive weight mappings for the two paths, and weighted fusion is performed to obtain the merged features. Finally Send in (includes) The detection module, which incorporates convolution, normalization, and activation functions, generates output features. Its gating mechanism can dynamically calibrate the contributions of the two paths, enhance the expressive power of the fused features, and the whole process relies on two multi-head attention modules to realize feature interaction, and completes feature fusion and optimization through subsequent steps to form a complete multi-scale feature fusion link; after identifying the goods, the identified goods are matched with the inventory results and the goods are settled.

[0030] A multi-dimensional hierarchical loss function for goods is constructed. Local features in the CNN backbone tend to encode local details, and excessive loss constraints may disrupt the representation of fine-grained features. Secondly, transformer and Mamba features possess higher and more global semantic abstraction, which can enhance inter-class separability. Furthermore, from a frequency perspective, low-frequency information strengthens the overall distribution, while high-frequency information has strong representation of edge details; different constraint methods can enhance the feature expression of different frequencies. Therefore, global features receive stronger discriminative guidance during gradient updates, while local features maintain a relatively stable geometric structure. The fixed marginal parameters for goods recognition are reconstructed into dynamic variables related to local and global feature branches. For example, if there are N classes of goods, the predefined goods metric W for each class can be expressed as: , in, The angle between the product feature vector and the target product category weight vector is calculated using the inverse cosine function. This angle directly reflects the similarity between the product features and the target category features in the feature space; a smaller angle indicates higher similarity. The target score is then obtained using the cosine function. The loss function is: , in, For loss function, As the baseline marginal parameter, These are global and local adjustment coefficients, ensuring that the marginal value exhibits a linear growth characteristic as the level of feature abstraction increases. The feature level depth parameter is used to distinguish between local and global features, corresponding to different feature extraction stages in the backbone network. For example, shallow features (local detail features) correspond to smaller depth parameters. Values, deep features (global semantic features) correspond to larger values Value, through The difference in values ​​enables differentiated loss constraints for features at different levels of abstraction. This is a frequency domain feature weight adjustment coefficient used to control the contribution strength of frequency domain features in the loss constraint. Combined with frequency and spatial domain adjustment coefficients, it can specifically strengthen the feature representation of low-frequency information (overall product distribution) or high-frequency information (product edge details), making the loss function more adaptable to the feature optimization needs of the frequency domain branch. These are the frequency domain and spatial domain adjustment coefficients. For the first Category of goods, The angle between the product feature vector and the weight vector of the target product category is denoted as . The dynamic marginal parameter is a function of the feature level depth d. Its core is to transform the fixed marginal parameter into a dynamic value related to the level of feature abstraction. Specifically, it consists of the baseline marginal parameter, global and local adjustment coefficients, and the feature level depth. Joint decision, For product categories, This is a category label for the product.

[0031] To verify the effectiveness of the MDFM network structure, ablation testing was performed on a self-made dataset. In each round of ablation learning, one or more modules were masked, while other parts remained unchanged. The recognition accuracy was used as the benchmark. Evaluate the model. This represents the probability that the first identified item is correctly labeled. See Table 1 for the results. Table 1

[0032] As shown in Table 1, the recognition and classification performance in real-world scenarios is significantly improved by combining a cross-domain, multi-angle backbone network with multi-angle hierarchical constraints. The data shows that Top1 improved from 75.2% of the baseline pure CNN network to 88.6%. When the two modules work together, the model's recognition rate is improved by 5 percentage points, which proves the effectiveness of combining cross-domain, multi-angle and metric learning constraints.

[0033] The results of the comparison with several recognition methods, including ViT-Arc (a combination of Transformer and arcface), ResNeXt-CE, and Res-SupCon, are shown in Table 2. Table 2

[0034] As shown in Table 2, MDFM achieves the best recognition rate for the top 1 indicator when identifying products.

[0035] In summary, the product recognition method for unmanned vending machines provided by this invention collects video data of users picking up goods using multi-view cameras in the vending machine. It then detects each frame of image data in the video data using a product image detection network to obtain product images. Finally, it recognizes the product images using a constructed MDFM network to obtain product recognition results. The MDFM network includes a multi-scale feature fusion module, a parallel Transformer module, a Mamba module, a frequency domain processing module, and a detection module. The Transformer module extracts global features from the product images, the Mamba module extracts contextual modeling features, the frequency domain processing module extracts edge detail features, the multi-scale feature fusion module fuses global features, contextual modeling features, and edge detail features to obtain merged features, and the detection module detects the merged features to obtain the product recognition result, thus improving the accuracy of product recognition.

[0036] To better implement the product identification method for unmanned vending machines in this embodiment of the invention, based on the product identification method for unmanned vending machines, the corresponding method is as follows: Figure 3 As shown, this embodiment of the invention also provides a product identification device for an unmanned vending machine. The product identification device 300 for the unmanned vending machine includes: Data acquisition module 301 is used to collect video data of users picking up goods through the multi-view camera of the unmanned vending machine; The product image acquisition module 302 is used to detect each frame of image data in the video data based on the product image detection network to obtain a product image; The product recognition module 303 is used to recognize product images based on the constructed MDFM network and obtain product recognition results. The MDFM network includes a multi-scale feature fusion module, a parallel Transformer module, a Mamba module, a frequency domain processing module, and a detection module. The Transformer module is used to extract global features of the product image, the Mamba module is used to extract context modeling features of the product image, the frequency domain processing module is used to extract edge detail features of the product image, the multi-scale feature fusion module is used to fuse global features, context modeling features, and edge detail features to obtain merged features, and the detection module is used to detect merged features to obtain product recognition results.

[0037] like Figure 4 As shown, the present invention also provides a product identification device 400, which can be a mobile terminal, desktop computer, laptop, handheld computer, server, or other computing device. The product identification device 400 includes a processor 401, a memory 402, and a display 403. Figure 4 Only some components of the product identification device 400 are shown; however, it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0038] In some embodiments, the memory 402 may be an internal storage unit of the product recognition device 400, such as a hard disk or memory of the product recognition device 400. In other embodiments, the memory 402 may be an external storage device of the product recognition device 400, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the product recognition device 400. Further, the memory 402 may include both internal and external storage units of the product recognition device 400. The memory 402 is used to store application software and various types of data installed on the product recognition device 400, such as the program code for installing the product recognition device 400. The memory 402 can also be used to temporarily store data that has been output or will be output. In one embodiment, the memory 402 stores a product recognition program for an unmanned vending machine, which can be executed by the processor 401 to implement the product recognition method for the unmanned vending machine according to various embodiments of the present invention.

[0039] In some embodiments, processor 401 may be a central processing unit (CPU), microprocessor or other data processing chip, used to run program code stored in memory 402 or process data, such as a product identification method for unmanned vending machines.

[0040] In some embodiments, display 403 may be an LED display, a liquid crystal display, a touch-screen liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. Display 403 is used to display identification information from the product identification program of the unmanned vending machine and to display a visual user interface. Components 401-403 of the product identification device 400 communicate with each other via a system bus.

[0041] In some embodiments, when the processor 401 executes the product identification program of the unmanned vending machine in the memory 402, it implements each step of the product identification method of the unmanned vending machine as described in the above embodiments. Since the product identification method of the unmanned vending machine has been described in detail above, it will not be repeated here.

[0042] Accordingly, the present invention also provides a computer-readable storage medium for storing a computer-readable program or instruction, which, when executed by a processor, can implement the steps or functions of the unmanned vending machine product identification method provided in the above-described method embodiments.

[0043] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0044] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for identifying goods in an unmanned vending machine, characterized in that, include: Video data of users picking up goods is collected through multi-angle cameras in unmanned vending machines; The product image is obtained by detecting each frame of image data in the video data using a product image detection network. The constructed MDFM network is used to identify the product image and obtain the product recognition result. The MDFM network includes a multi-scale feature fusion module, a parallel Transformer module, a Mamba module, a frequency domain processing module, and a detection module. The Transformer module is used to extract global features of the product image, the Mamba module is used to extract context modeling features of the product image, the frequency domain processing module is used to extract edge detail features of the product image, the multi-scale feature fusion module is used to fuse the global features, the context modeling features, and the edge detail features to obtain merged features, and the detection module is used to detect the merged features to obtain the product recognition result.

2. The product identification method for unmanned vending machines according to claim 1, characterized in that, The product image detection network includes a backbone network, a neck network, and a head network. The process of detecting each frame of image data in the video data using the product image detection network to obtain a product image includes: Based on the backbone network, multi-scale feature extraction is performed on each frame of image data in the video data to obtain first image features, second image features, third image features and fourth image features; The neck feature pyramid based on the neck network fuses the first image feature, the second image feature, the third image feature, and the fourth image feature to obtain three-scale features; The three-scale features are classified based on the classification branch of the head network to obtain product images, non-product images, and product categories. The target boundary of the product image is corrected by the regression branch of the head network to determine the location of the product. The non-product images include face images.

3. The product identification method for unmanned vending machines according to claim 2, characterized in that, The backbone network includes a first basic residual block, a second basic residual block, a first bottleneck residual block, and a second bottleneck residual block; based on the backbone network, multi-scale feature extraction is performed on each frame of image data in the video data to obtain a first image feature, a second image feature, a third image feature, and a fourth image feature, including: Shallow feature extraction is performed on each frame of image data based on the first basic residual block and the second basic residual block to obtain the first image feature and the second image feature. The first basic residual block and the second basic residual block have the same structure. The first basic residual block includes a convolutional layer, a batch normalization layer, an activation function, and a residual connection. Based on the first bottleneck residual block and the second bottleneck residual block, deep feature extraction is performed on each frame of image data to obtain the third image feature and the fourth image feature.

4. The product identification method for unmanned vending machines according to claim 2, characterized in that, The multi-scale feature fusion module includes a first multi-head attention module and a second multi-head attention module; the product image recognition based on the constructed MDFM network to obtain product recognition results includes: The product image is input into the MDFM network, and the product image is used to extract features through the shared convolution of the MDFM network to obtain low-level features; The underlying features are input into the Transformer module, the Mamba module, and the frequency domain processing module respectively to obtain global features, context modeling features, and edge detail features. Convolution projection is then performed on the global features, context modeling features, and edge detail features to obtain a first feature vector sequence, a second feature vector sequence, and a third feature vector sequence. Based on the first multi-head attention module, the first feature vector sequence, the second feature vector sequence, and the third feature vector sequence are spatially aligned to obtain global spatial features; Based on the second multi-head attention module, the first feature vector sequence, the second feature vector sequence, and the third feature vector sequence are concatenated along the channel dimension to obtain global channel features; A gated fusion mechanism is used to perform weighted fusion of the global spatial features and global channel features to obtain merged features; The detection module detects the merged features to obtain the product identification result.

5. The product identification method for unmanned vending machines according to claim 4, characterized in that, The Transformer module includes a block embedding layer, a depthwise separable convolutional layer, a normalization layer, an activation layer, a first ViT module, a second ViT module, a third ViT module, and a convolutional layer.

6. The product identification method for unmanned vending machines according to claim 4, characterized in that, The Mamba module includes a first VSS module, a second VSS module, a third VSS module, and a convolutional layer; the frequency domain processing module includes a Fourier transform layer, a separation layer, a frequency domain filtering layer, a frequency domain network module, an inverse Fourier transform layer, a convolutional layer, and residual connections; the detection module includes a convolutional layer, a normalization layer, and an activation layer.

7. The product identification method for unmanned vending machines according to claim 4, characterized in that, The loss function of the MDFM network is: ， in, For loss function, As the baseline marginal parameter, These are the global and local adjustment coefficients. For feature level depth parameters, For frequency domain feature weight adjustment coefficients, These are the frequency domain and spatial domain adjustment coefficients. For the first Category of goods, The angle between the product feature vector and the weight vector of the target product category is denoted as . For dynamic marginal parameters, For product categories, This is the category label for the product.

8. A product identification device for an unmanned vending machine, characterized in that, include: The data acquisition module is used to collect video data of users picking up goods through the multi-view cameras of the unmanned vending machine; The product image acquisition module is used to detect each frame of image data in the video data based on the product image detection network to obtain a product image; The product recognition module is used to recognize the product image based on the constructed MDFM network to obtain the product recognition result. The MDFM network includes a multi-scale feature fusion module, a parallel Transformer module, a Mamba module, a frequency domain processing module, and a detection module. The Transformer module is used to extract global features of the product image, the Mamba module is used to extract context modeling features of the product image, the frequency domain processing module is used to extract edge detail features of the product image, the multi-scale feature fusion module is used to fuse the global features, the context modeling features, and the edge detail features to obtain merged features, and the detection module is used to detect the merged features to obtain the product recognition result.

9. A product identification device, characterized in that, Including memory and processor; The memory stores a computer-readable program that can be executed by the processor; When the processor executes the computer-readable program, it implements the steps of the product identification method for unmanned vending machines as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, Used to store computer-readable programs or instructions, which, when executed by a processor, can implement the steps in the product identification method of the unmanned vending machine as described in any one of claims 1-7.

Images