A method and device for intelligent product recognition in unmanned vending machines, from coarse to fine.

Abstract

This invention discloses a coarse-to-fine intelligent product recognition method and apparatus for unmanned vending machines. The method includes: dividing a product image into coarse-scale grid blocks and fine-scale grid blocks, and generating coarse-level and fine-level grid block representations; inputting the coarse-level grid block representations into a Transformer encoder to obtain coarse-level global features, and adaptively acquiring a coarse-scale product region mask on the coarse-scale grid blocks; based on the coarse-scale product region mask, selecting fine-scale grid blocks within the product region and their corresponding position embeddings from the fine-scale grid blocks, and calculating the fine-level representation embedding of the product region; obtaining fine-level product features based on the fine-level representation embedding; fusing the coarse-scale grid block representations and fine-level product features to obtain fused features; and inputting the fused features into a classifier to obtain the product classification result. This invention can significantly reduce computational complexity while minimizing interference from invalid background information, thereby improving product recognition accuracy.

Description

Technical Field

[0001] This invention relates to the field of product recognition technology, and in particular to a method, device, storage medium, and electronic device for intelligent product recognition in unmanned vending machines, ranging from coarse to fine. Background Technology

[0002] Product recognition is a significant challenge in the field of computer vision, and solving this problem will greatly promote the development of unmanned intelligent vending machines. In recent years, with the development of deep learning technology, especially the introduction of the Transformer framework, significant progress has been made in product recognition tasks. With the promotion and popularization of unmanned vending machines as a new type of intelligent retail, higher demands are being placed on the efficiency and accuracy of product recognition technology.

[0003] Currently, mainstream product recognition methods all employ deep learning networks to extract high-dimensional features, and then use loss functions such as triples or ArcFace to constrain these features during training, ensuring that features of the same type of product are similar and features of different types of products are dissimilar. The core technology in existing methods lies in how to extract high-dimensional features from product images with strong expressive power. Before the advent of the Transformer framework, the most commonly used framework was the Convolutional Neural Network (CNN). While CNNs can effectively extract local features of the target, they neglect the global correlations within the image, resulting in limited product recognition accuracy. The Transformer framework, on the other hand, fully utilizes the self-attention mechanism, effectively capturing long-distance dependencies in images and extracting features with stronger expressive power.

[0004] Currently, the Transformer has become the mainstream framework in the field of deep learning and the foundational framework for various large-scale models. However, because the computational cost of the self-attention mechanism is exponentially related to the number of input image patches, the Transformer's computational cost is extremely high, making it difficult to meet the requirements of high-efficiency product recognition. To address this issue, one approach is to reduce the number of input image patches, but this leads to excessively large scales during feature extraction, making it difficult to extract refined product features. Another approach is to limit the scope of self-attention computation, performing the self-attention mechanism in local regions, but this disrupts long-distance contextual features, resulting in a decrease in the expressive power of the extracted features. In addition to its high computational cost, the self-attention mechanism in the Transformer is also prone to associating with a large amount of invalid information. Furthermore, the product images captured by vending machines contain a large amount of complex and invalid background information, which greatly interferes with the extraction and expression of effective product features, thereby reducing the accuracy of product recognition. Moreover, if invalid background regions also participate in the self-attention computation, it will significantly increase the computational cost. Summary of the Invention

[0005] This invention provides a method, device, storage medium, and electronic device for intelligent product recognition in unmanned vending machines, which can significantly reduce computational complexity while minimizing interference from invalid background information and improving product recognition accuracy.

[0006] This invention provides a coarse-to-fine intelligent product recognition method for unmanned vending machines, comprising: Acquire product images; The coarse-level grid block representation is input into the Transformer encoder to obtain coarse-level global features, and a coarse-scale commodity region mask is adaptively obtained on the coarse-scale grid block. Based on the coarse-scale product region mask, the fine-scale grid blocks in the product region and their corresponding position embeddings are selected from the fine-scale grid blocks, and the fine-level representation embedding of the product region is calculated. Based on the fine-level representation embedding, the fine-level product features are obtained. Based on the coarse-scale product region mask, the grid blocks in the product region and their corresponding position embeddings are selected from the fine-scale grid blocks, and the fine-level representation embedding of the product region is calculated. Based on the fine-level representation embedding, the fine-level product features are obtained. The coarse-scale grid block sequence is downsampled to obtain a coarse-scale grid block representation. The coarse-scale grid block representation is then fused with the fine-level commodity features to obtain a fused feature. The fused features are input into the classifier to obtain the product classification result.

[0007] Furthermore, according to the above-described coarse-to-fine intelligent product recognition method for unmanned vending machines, the product image is divided into coarse-scale grid blocks and fine-scale grid blocks. Based on the coarse-scale grid blocks, coarse-level grid block representations and coarse-level grid block representation embeddings are generated. Based on the fine-scale grid blocks, fine-level grid block representations and fine-level grid block representation embeddings are generated, including: The product image is segmented and flattened into multiple coarse-scale two-dimensional grid blocks. The two-dimensional grid blocks are linearly projected to obtain coarse-level grid block representations. Multiple coarse-level grid block representations form a coarse-scale grid block feature sequence. Adding global representations and learnable positional embeddings to the coarse-scale grid block feature sequence yields a coarse-level grid block representation embedding. Linear projection is performed on the fine-scale grid block to obtain a fine-level grid block representation, and multiple fine-level grid block representations form a fine-scale grid block feature sequence. A fine-level grid block representation embedding is obtained by adding a global representation and a learnable positional embedding to the fine-scale grid block feature sequence.

[0008] Furthermore, according to the above-described coarse-to-fine intelligent product recognition method for unmanned vending machines, the coarse-level grid block representation is input into the Transformer encoder to obtain coarse-level global features, including: A query vector is generated based on the global representation, and a key matrix and a value matrix are generated based on the coarse-scale grid block feature sequence. The query vector and the key matrix are interacted to calculate the attention weight matrix for each coarse-scale grid block; Based on the attention weight matrix and the value matrix, a coarse-level global feature based on self-attention is generated.

[0009] Furthermore, according to the above-described coarse-to-fine intelligent product recognition method for vending machines, the adaptive acquisition of a coarse-scale product region mask on the coarse-scale grid block includes: Calculate the average attention weight matrix of all attention heads in the Transformer encoder based on the aforementioned attention weight matrix; Search for the coarse-scale grid blocks corresponding to the first few largest values ​​in the average attention weight matrix and mark them as 1. Mark the remaining coarse-scale grid blocks as 0 to obtain the coarse-scale product region mask.

[0010] Furthermore, according to the above-described coarse-to-fine intelligent product recognition method for vending machines, the fine-scale grid blocks within the product area and their corresponding position embeddings are selected from the fine-scale grid blocks based on the coarse-scale product area mask, and the fine-level representation embedding of the product area is calculated using the following formula:

[0011]

[0012]

[0013] in, To select the precise-scaled grid blocks located within the product area mask, For coarse-scale commodity area masking, Embedding of high-level grid block representations, Embed at the corresponding position of the grid block in the product area. Position embeddings are added to the feature sequences of fine-scale grid blocks. For high-level representation embedding, This is a global representation.

[0014] Furthermore, according to the above-described coarse-to-fine intelligent identification method for unmanned vending machines, the fine-level product features obtained based on the fine-level representation embedding include: The refined hierarchical representation is embedded into the Transformer encoder to construct a self-attention-based refined hierarchical product feature, expressed by the following formula:

[0015] in, As a characteristic of high-end products, This represents the query vector generated by the high-level global representation. and These represent the key matrix and value matrix, respectively, obtained by embedding the refined hierarchical representation of the commodity region through a linear projection layer. Here, D represents the feature dimension of the query vector, which is the activation function.

[0016] Furthermore, according to the above-described coarse-to-fine intelligent product recognition method for unmanned vending machines, the coarse-scale grid block representation is fused with the fine-level product features to obtain a fused feature, which is expressed by the following formula:

[0017] in, As a feature of fusion, This represents the query vector generated by the high-level global representation. and These represent the key matrix and value matrix, respectively, generated by the linear projection layer from the coarse-scale grid block representation.

[0018] This invention also provides a coarse-to-fine intelligent product recognition device for unmanned vending machines, comprising: The acquisition module is used to acquire product images; The grid division and representation generation module is used to divide the commodity image into coarse-scale grid blocks and fine-scale grid blocks, generate coarse-level grid block representations and coarse-level grid block representation embeddings based on the coarse-scale grid blocks, and generate fine-level grid block representations and fine-level grid block representation embeddings based on the fine-scale grid blocks. The coarse-level feature extraction and mask generation module is used to input the coarse-level grid block representation into the Transformer encoder to obtain coarse-level global features, and adaptively obtain a coarse-scale commodity region mask on the coarse-scale grid block. The fine-scale feature extraction module is used to filter out the fine-scale grid blocks in the product region and their corresponding position embeddings from the fine-scale grid blocks based on the coarse-scale product region mask, calculate the fine-scale representation embedding of the product region, and obtain the fine-scale product features based on the fine-scale representation embedding. The fusion module is used to downsample the coarse-scale grid block sequence to obtain a coarse-scale grid block representation, and then fuse the coarse-scale grid block representation with the fine-level commodity features to obtain a fused feature. The classification module is used to input the fused features into the classifier to obtain the product classification result.

[0019] The present invention also provides a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to execute any of the above-described coarse-to-fine intelligent identification methods for unmanned vending machines.

[0020] The present invention also provides an electronic device, including a processor and a memory, wherein the processor is electrically connected to the memory, the memory is used to store instructions and data, and the processor is used in the steps of the coarse-to-fine intelligent identification method for unmanned vending machines described in any of the preceding claims.

[0021] This invention provides a method, device, storage medium, and electronic device for intelligent product recognition in unmanned vending machines, ranging from coarse to fine scales. By constructing an innovative two-level feature processing mechanism, this invention aims to significantly reduce computational complexity while improving recognition accuracy. Specifically, the invention first divides the input product image into coarse-scale and fine-scale grid blocks according to different resolutions, thereby achieving multi-scale grid division of the image. Then, based on the coarse-scale grid blocks, the invention uses a Transformer encoder to extract global features at the coarse level and further uses a product effective region localization module to adaptively filter out product background information, preserving as much of the effective product region as possible for final recognition. Since the number of image blocks input at the coarse-grained level is small, the self-attention computation at this level is relatively low. Subsequently, guided by the product region localization results, the invention uses a Transformer encoder to extract product features at the fine scale. In this step, the fine-grained self-attention computation is limited to the effective product region, which reduces computational load and avoids interference from complex background invalid information. Finally, this invention utilizes a cross-level feature interaction module to integrate coarse-level global features into fine-level product features. This reduces computational costs while compensating for the limited receptive field of the fine-level network, resulting in fused features that encompass both global image data and local product regions, which are then used for product recognition. This invention innovatively constructs a coarse-to-fine two-level product target recognition mechanism, adaptively locating the most relevant effective region for the product recognition task and achieving cross-level feature interaction and fusion. This significantly reduces computational complexity while minimizing interference from invalid background information, thereby improving product recognition accuracy. Attached Figure Description

[0022] The technical solution and other beneficial effects of the present invention will become apparent from the following detailed description of specific embodiments of the invention, in conjunction with the accompanying drawings.

[0023] Figure 1A flowchart of a coarse-to-fine intelligent product recognition method for unmanned vending machines provided in an embodiment of the present invention.

[0024] Figure 2 This is a flowchart for obtaining a coarse-scale commodity area mask provided in an embodiment of the present invention.

[0025] Figure 3 A flowchart for obtaining fusion features provided in an embodiment of the present invention.

[0026] Figure 4 The image shows a visual comparison of the accuracy and computational efficiency of the method provided in this embodiment of the invention with other methods on the RP2K dataset.

[0027] Figure 5 This is a structural diagram of the intelligent product recognition device for unmanned vending machines, which is designed for coarse-to-fine identification, provided in an embodiment of the present invention.

[0028] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] In product recognition tasks, existing methods generally suffer from the following technical problems: First, the traditional Transformer architecture processes all regions in the image equally, resulting in a large amount of computing resources being wasted on irrelevant background areas; second, there is a lack of an effective screening mechanism for features relevant to the recognition task, making it difficult to maintain recognition accuracy while reducing computing costs.

[0031] To address the aforementioned problems, embodiments of the present invention provide a method, apparatus, storage medium, and electronic device for coarse-to-precise intelligent identification of goods in unmanned vending machines. The coarse-to-precise intelligent identification device for goods in unmanned vending machines provided by embodiments of the present invention can be integrated into an electronic device, which can be a terminal, server, or other device. The terminal can include a tablet computer, laptop computer, personal computer (PC), microprocessor box, or other devices.

[0032] Please see Figure 1 , Figure 1The flowchart illustrates a coarse-to-fine intelligent product recognition method for vending machines provided in this embodiment of the invention. This method, applied in electronic devices, includes the following steps: S1, acquire product image.

[0033] S2 divides the product image into coarse-scale grid blocks and fine-scale grid blocks. Based on the coarse-scale grid blocks, coarse-level grid block representations and coarse-level grid block representation embeddings are generated. Based on the fine-scale grid blocks, fine-level grid block representations and fine-level grid block representation embeddings are generated.

[0034] The input product image is divided into coarse-scale and fine-scale grids. To reduce the computational cost of the Transformer feature encoding process, this invention employs a two-level feature processing mechanism, dividing the product image to be identified into coarse-scale and fine-scale grid blocks as multi-scale inputs to the model, laying the foundation for subsequent extraction of coarse-level global features and fine-level product features.

[0035] In one embodiment, step S2 includes the following steps: S21, the product image is segmented and flattened into multiple coarse-scale two-dimensional grid blocks. The two-dimensional grid blocks are linearly projected to obtain coarse-level grid block representations. Multiple coarse-level grid block representations form a coarse-scale grid block feature sequence.

[0036] Specifically, the present invention segments and flattens a two-dimensional image into... Each coarse-scale two-dimensional grid block is further transformed through a linear projection layer to obtain its corresponding initial grid block representation, and these representations are then merged to generate a set of coarse-scale grid block feature sequences. This operation can be expressed by the following formula:

[0037] in, Indicates the first i individual grid blocks D 3D characterization, and This represents linear projection layers and gridding operations at a coarse-scale level. This represents the input product image.

[0038] S22, global representation and learnable positional embedding are added to the coarse-scale grid block feature sequence to obtain coarse-level grid block representation embedding.

[0039] Building upon this foundation, this invention further introduces a global [CLS] representation, which is added before the grid patch embedding vector to aggregate the contextual information of the entire image. The global [CLS] representation is a learnable tensor vector. Furthermore, all grid patch representations are augmented with a learnable positional embedding to preserve the spatial location information of the input grid patches. This process can be formulated as follows:

[0040] in, This represents the coarse-level global [CLS] representation. This represents the set of positions embedded in each coarse-scale grid block. The final coarse-level grid block representation embedding is represented.

[0041] S23, linear projection is performed on the fine-scale grid blocks to obtain fine-level grid block representations, and multiple fine-level grid block representations form a fine-scale grid block feature sequence.

[0042] S24, global representation and learnable positional embedding are added to the fine-scale grid block feature sequence to obtain the fine-level grid block representation embedding.

[0043] To preserve local details of the product, this invention further subdivides each coarse-scale grid block into 2×2 sub-grid blocks, and obtains a result containing [details omitted]. A sequence of fine-scale grid blocks Subsequently, a global [CLS] representation was introduced. and location information embedding Generate high-level grid block representation embeddings This process can be expressed by the following formula:

[0044]

[0045]

[0046] S3 inputs the coarse-level grid block representation into the Transformer encoder to obtain coarse-level global features, and adaptively obtains the coarse-scale commodity region mask on the coarse-scale grid block.

[0047] It is worth noting that because the computational complexity of the self-attention mechanism in Transformer is exponentially related to the number of output grid blocks, choosing coarse-level grid blocks (a smaller number) for feature encoding can significantly reduce computational costs. Furthermore, to avoid interference from irrelevant background information, this invention designs a product region recognition module to adaptively locate information-rich product regions, providing support for subsequent fine-level product feature extraction and cross-level feature interaction.

[0048] In one embodiment, step S3, inputting the coarse-level grid block representation into the Transformer encoder to obtain coarse-level global features, includes the following steps: S31 generates query vectors based on global representations and generates key and value matrices based on coarse-scale grid block feature sequences.

[0049] S32, the query vector and key matrix are interacted to calculate the attention weight matrix for each coarse-scale grid block.

[0050] S33 generates coarse-level global features based on self-attention, using the attention weight matrix and value matrix.

[0051] The coarse-level grid block representations are input into the Transformer encoder, and the self-attention mechanism promotes the information interaction between the global [CLS] representation and other grid block representations, so as to achieve the purpose of aggregating global context information from the input sequence.

[0052] Specifically, firstly, a query vector is generated based on the global [CLS] representation using a multilayer perceptron, and simultaneously based on the feature sequence of coarse-scale grid blocks. Generate key matrix Sum matrix Then, the query vector will be... AND key matrix Interact to calculate the attention weight matrix for each grid block, and adjust the value matrix. Perform linear combination to generate self-attention-based... This process can be formalized as follows:

[0053] in, Represents a non-linear activation function. D This represents the feature dimension of the query vector. This represents the attention weight matrix.

[0054] Considering the presence of many invalid interference regions such as background in images, not all image regions contribute meaningfully to the final product recognition. Indiscriminately performing feature interactions on all grid blocks inevitably incurs high computational costs and aggregates many invalid features, interfering with the extraction of final product recognition features. To overcome this technical deficiency, this invention designs a more efficient scheme: adaptively identifying and locating the regions most relevant to the product while discarding interference from irrelevant background information, thereby improving the model's recognition efficiency and accuracy.

[0055] In one embodiment, Figure 2 A flowchart for obtaining a coarse-scale commodity region mask provided in an embodiment of the present invention is shown below. Figure 2 As shown, step S3, adaptively obtaining the coarse-scale product region mask on the coarse-scale grid block, includes the following steps: S34, calculate the average attention weight matrix of all attention heads in the Transformer encoder based on the attention weight matrix.

[0056] Specifically, based on the attention distribution matrix This is used to identify the most critical coarse-scale grid blocks, laying the foundation for subsequent product region recognition. The average attention weight matrix of all attention heads in the Transformer encoder is first calculated using the following formula. :

[0057] in, Indicates the first h The attention distribution matrix associated with global features in each attention head. H This represents the total number of attention heads.

[0058] S35: Search for the coarse-scale grid blocks corresponding to the first few largest values ​​in the average attention weight matrix and label them as 1. Label the remaining coarse-scale grid blocks as 0 to obtain the coarse-scale product region mask.

[0059] First, based on the average attention weight matrix obtained in the previous step, search... Center front K The coarse-scale grid block corresponding to the maximum value The most relevant area to the product is marked as 1, while the remaining grid cells are marked as 0, thus obtaining a coarse-scale product area mask. This process can be expressed by the following formula:

[0060]

[0061]

[0062] in, ( ) indicates the preceding K A search function for the maximum value. This represents the value of each coarse-scale grid block in the mask. K The smaller the value of , the more grid blocks with low attention scores will be discarded (the mask area of ​​the product region will be smaller), thus significantly reducing the computational cost of subsequent feature interactions.

[0063] S4: Based on the coarse-scale product region mask, select the fine-scale grid blocks in the product region and their corresponding position embeddings from the fine-scale grid blocks, calculate the fine-level representation embedding of the product region, and obtain the fine-level product features based on the fine-level representation embedding.

[0064] At a coarse scale, more global features of the image can be obtained, but due to the large grid scale, it is difficult to extract effective local features of the product. In order to preserve finer local details of the product area, this invention further extracts product features at a fine scale, guided by the adaptive localization results of the product area.

[0065] Based on the identified valid product area (masked) (This is an example of how the present invention filters out grid blocks located in the product area from a globally scaled, finely sized grid.) and its corresponding position embedding Then, the fine-level representation embedding of the product region is calculated. This process can be expressed by the following formula:

[0066]

[0067]

[0068] in, To select the precise-scaled grid blocks located within the product area mask, For coarse-scale commodity area masking, Embedding of high-level grid block representations, Embed at the corresponding position of the grid block in the product area. Position embeddings are added to the feature sequences of fine-scale grid blocks. For high-level representation embedding, This is a global representation.

[0069] To further facilitate information exchange between fine-scale grid blocks, this invention embeds fine-level representations. The input is fed into the Transformer encoder to construct high-level product features based on self-attention. This process can be formalized as follows:

[0070] in, As a characteristic of high-end products, This represents the query vector generated by the high-level global representation. and These represent the key matrix and value matrix, respectively, obtained by embedding the refined hierarchical representation of the commodity region through a linear projection layer. Here, D represents the feature dimension of the query vector, which is the activation function.

[0071] S5. The coarse-scale grid block sequence is downsampled to obtain the coarse-scale grid block representation. The coarse-scale grid block representation is then fused with the fine-level commodity features to obtain the fused features.

[0072] By integrating coarse-grained global features into fine-grained product features, the problem of limited receptive field in fine-scale networks is effectively compensated for, resulting in fused features that combine global image data with local product region data, which are then used for product recognition. Specifically, Figure 3 A flowchart for obtaining fusion features provided in embodiments of the present invention, such as Figure 3 As shown. This process enables each fine-scale grid block to exchange information with all coarse-scale grid blocks through the fine-level global [CLS] representation, thereby incorporating global context dependencies into the fine-level representation.

[0073] To further improve computational efficiency, this invention performs lightweight downsampling on the coarse-scale grid block sequence, generating a set of fewer representative grid block embeddings. Specifically, this invention uses deep convolutional layers to reduce the total number of coarse-scale grid block sequences from... downsampling to This reduces the computational cost of subsequent feature interactions and achieves lower overall complexity. This process can be formulated as follows:

[0074] in, This represents a downsampling operation based on depthwise convolution. This represents the coarse-scale grid block representation after sampling.

[0075] By employing a cross-attention mechanism, the global information represented by the downsampled coarse-scale grid blocks is fused into the fine-level product features, thereby obtaining a fused feature that combines local details with global information. This process can be formalized as follows:

[0076] in, As a feature of fusion, This represents the query vector generated by the high-level global representation. and These represent the key matrix and value matrix, respectively, generated by the linear projection layer from the coarse-scale grid block representation.

[0077] Given that the number of fine-scale grid blocks is relatively small, and the number of coarse-scale grid blocks after downsampling is also limited, the computational complexity of the cross-attention operation is greatly reduced, and the computational efficiency is significantly improved.

[0078] S6. Input the fused features into the classifier to obtain the product classification result.

[0079] This invention integrates detailed information of the product area and global context information into a fusion feature. The data is input into a classifier to obtain confidence scores for each category, and the product category corresponding to the highest score is found, thus completing the product recognition task.

[0080] This invention was tested on three publicly available datasets (Products-10K, RP2K, and Retail1K) and compared with currently widely used methods. Table 1 shows the comparison results of the recognition accuracy and computational efficiency of the method provided in this embodiment of the invention with other methods on the three publicly available datasets.

[0081] Table 1. Comparison of recognition accuracy and computational efficiency between this method and other methods on three public datasets.

[0082] This invention achieved optimal recognition accuracy and computational efficiency on three datasets, fully validating its outstanding performance in product recognition tasks. Furthermore, Figure 4 This is a visualization comparing the accuracy and computational efficiency of the method provided in this embodiment of the invention with other methods on the RP2K dataset. Figure 4 This more intuitively reflects the technical superiority of the present invention.

[0083] Based on the method described in the above embodiments, this embodiment will further describe the unmanned vending machine product intelligent recognition device from the perspective of coarse to fine. The unmanned vending machine product intelligent recognition device from coarse to fine can be implemented as an independent entity or integrated into an electronic device. The electronic device can be a terminal, server or other device. The terminal can include a tablet computer, a laptop computer, a personal computer (PC), a microprocessor box, or other devices.

[0084] Please see Figure 5 , Figure 5 This invention specifically describes a coarse-to-fine intelligent product recognition device for vending machines, applicable to electronic devices. This coarse-to-fine intelligent product recognition device may include: The acquisition module is used to acquire product images; The grid division and representation generation module is used to divide the commodity image into coarse-scale grid blocks and fine-scale grid blocks, generate coarse-level grid block representations and coarse-level grid block representation embeddings based on the coarse-scale grid blocks, and generate fine-level grid block representations and fine-level grid block representation embeddings based on the fine-scale grid blocks. The coarse-level feature extraction and mask generation module is used to input the coarse-level grid block representation into the Transformer encoder to obtain coarse-level global features, and adaptively obtain a coarse-scale commodity region mask on the coarse-scale grid block. The fine-scale feature extraction module is used to filter out the fine-scale grid blocks in the product region and their corresponding position embeddings from the fine-scale grid blocks based on the coarse-scale product region mask, calculate the fine-scale representation embedding of the product region, and obtain the fine-scale product features based on the fine-scale representation embedding. The fusion module is used to downsample the coarse-scale grid block sequence to obtain a coarse-scale grid block representation, and then fuse the coarse-scale grid block representation with the fine-level commodity features to obtain a fused feature. The classification module is used to input the fused features into the classifier to obtain the product classification result.

[0085] In specific implementation, the above modules and / or units can be implemented as independent entities, or they can be arbitrarily combined and implemented as the same or several entities. For the specific implementation of the above modules and / or units, please refer to the previous method embodiments. For the specific beneficial effects that can be achieved, please also refer to the beneficial effects in the previous method embodiments, which will not be repeated here.

[0086] In addition, this embodiment of the invention also provides an electronic device, which may be a computer, tablet computer, or other similar device. This electronic device can implement the steps of any embodiment of the coarse-to-fine intelligent product recognition method for unmanned vending machines provided by this invention. Therefore, it can achieve the beneficial effects achievable by any coarse-to-fine intelligent product recognition method for unmanned vending machines provided by this invention, as detailed in the preceding embodiments, and will not be repeated here.

[0087] Figure 6The diagram illustrates a specific structural block diagram of an electronic device provided in an embodiment of the present invention. This electronic device can be used to implement the coarse-to-fine intelligent identification method for unmanned vending machines provided in the above embodiments. The electronic device 500 can be a terminal, server, or other device. The terminal can include a tablet computer, laptop computer, personal computer (PC), microprocessor box, or other devices.

[0088] The memory 520 can be used to store software programs and modules, such as the program instructions / modules corresponding to those in the above embodiments. The processor 580 executes various functional applications and data processing by running the software programs and modules stored in the memory 520, such as taking pictures with the front-facing camera, processing the captured images, and switching the display colors of the content displayed on the screen. The memory 520 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 520 may further include memory remotely located relative to the processor 580, and these remote memories can be connected to the electronic device 500 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0089] The input unit 530 can be used to receive input numeric or character information, and to generate a keyboard and mouse related to user settings and function control.

[0090] Display unit 540 can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces, which can be composed of graphics, text, icons, video, and any combination thereof. Display unit 540 may include display panel 541, which may optionally be configured in the form of LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), or other similar forms.

[0091] Electronic device 500, through transmission module 570 (e.g., Wi-Fi module), can help users receive requests, send information, etc., providing users with wireless broadband internet access. Although transmission module 570 is shown in the figure, it is understood that it is not an essential component of electronic device 500 and can be omitted as needed without changing the essence of the invention.

[0092] The processor 580 is the control center of the electronic device 500. It connects to various parts of the phone via various interfaces and lines, and performs various functions and processes data of the electronic device 500 by running or executing software programs and / or modules stored in the memory 520, and by calling data stored in the memory 520, thereby providing overall monitoring of the electronic device. Optionally, the processor 580 may include one or more processing cores; in some embodiments, the processor 580 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into the processor 580.

[0093] Electronic device 500 also includes a power supply 590 (such as a battery) that supplies power to various components. In some embodiments, the power supply may be logically connected to processor 580 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 590 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0094] Although not shown, the electronic device 500 also includes cameras (such as front-facing cameras and rear-facing cameras), Bluetooth modules, etc., which will not be described in detail here. Specifically, in this embodiment, the display unit of the electronic device is a touch screen display, and the mobile terminal also includes a memory and one or more programs, wherein one or more programs are stored in the memory and configured to be executed by one or more processors. One or more programs contain instructions for performing the following operations: Acquire product images; The product image is divided into coarse-scale grid blocks and fine-scale grid blocks. Coarse-level grid block representations and coarse-level grid block representation embeddings are generated based on the coarse-scale grid blocks. Fine-level grid block representations and fine-level grid block representation embeddings are generated based on the fine-scale grid blocks. The coarse-level grid block representation is input into the Transformer encoder to obtain coarse-level global features, and a coarse-scale commodity region mask is adaptively obtained on the coarse-scale grid block. Based on the coarse-scale product region mask, the fine-scale grid blocks in the product region and their corresponding position embeddings are selected from the fine-scale grid blocks, and the fine-level representation embedding of the product region is calculated. Based on the fine-level representation embedding, the fine-level product features are obtained. The coarse-scale grid block sequence is downsampled to obtain a coarse-scale grid block representation. The coarse-scale grid block representation is then fused with the fine-level commodity features to obtain a fused feature. The fused features are input into the classifier to obtain the product classification result.

[0095] In practice, the above modules can be implemented as independent entities or combined in any way to be implemented as the same or several entities. For the specific implementation of the above modules, please refer to the previous method implementation examples, which will not be repeated here.

[0096] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor. Therefore, embodiments of the present invention provide a storage medium storing multiple instructions that can be loaded by a processor to execute the steps of any embodiment of the coarse-to-fine intelligent product recognition method for unmanned vending machines provided by the present invention.

[0097] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0098] Since the instructions stored in the storage medium can execute the steps in any embodiment of the coarse-to-fine intelligent identification method for unmanned vending machines provided in the embodiments of the present invention, the beneficial effects that the coarse-to-fine intelligent identification method for unmanned vending machines provided in the embodiments of the present invention can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0099] The foregoing has provided a detailed description of a method, apparatus, storage medium, and electronic device for intelligent identification of goods in unmanned vending machines, from coarse to fine, as provided in the embodiments of the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for intelligent product recognition in unmanned vending machines, characterized in that, The method includes: Acquire product images; A two-level feature processing mechanism is adopted to divide the commodity image into coarse-scale grid blocks and fine-scale grid blocks. Based on the coarse-scale grid blocks, coarse-level grid block representations and coarse-level grid block representation embeddings are generated. Based on the fine-scale grid blocks, fine-level grid block representations and fine-level grid block representation embeddings are generated. The coarse-level grid block representation is input into the Transformer encoder to obtain coarse-level global features, and a coarse-scale commodity region mask is adaptively obtained on the coarse-scale grid block. Adaptively obtaining a coarse-scale product region mask on coarse-scale grid blocks includes: calculating the average attention weight matrix of all attention heads in the Transformer encoder based on the attention weight matrix output by the Transformer encoder; searching for the coarse-scale grid blocks corresponding to the first few largest values ​​in the average attention weight matrix and marking them as 1, while marking the remaining coarse-scale grid blocks as 0, to obtain the coarse-scale product region mask; expressed by the following formula: in, ( ) indicates the preceding K A search function for the maximum value. This represents the value of each coarse-scale grid block in the mask. The average attention weight matrix for all attention heads. for Center front K The coarse-scale grid block corresponding to the largest value. For coarse-scale product area masking; Based on the coarse-scale product region mask, the fine-scale grid blocks in the product region and their corresponding position embeddings are selected from the fine-scale grid blocks, and the fine-level representation embedding of the product region is calculated. Based on the fine-level representation embedding, the fine-level product features are obtained. The coarse-scale grid block sequence is downsampled to obtain a coarse-scale grid block representation. The coarse-scale grid block representation is then fused with the fine-level commodity features through a cross-attention mechanism to obtain a fused feature. The fused features are input into the classifier to obtain the product classification result.

2. The method for intelligent product recognition in unmanned vending machines from coarse to fine according to claim 1, characterized in that, The product image is divided into coarse-scale grid blocks and fine-scale grid blocks. Coarse-level grid block representations and coarse-level grid block representation embeddings are generated based on the coarse-scale grid blocks. Fine-level grid block representations and fine-level grid block representation embeddings are generated based on the fine-scale grid blocks, including: The product image is segmented and flattened into multiple coarse-scale two-dimensional grid blocks. The two-dimensional grid blocks are linearly projected to obtain coarse-level grid block representations. Multiple coarse-level grid block representations form a coarse-scale grid block feature sequence. Adding global representations and learnable positional embeddings to the coarse-scale grid block feature sequence yields a coarse-level grid block representation embedding. Linear projection is performed on the fine-scale grid block to obtain a fine-level grid block representation, and multiple fine-level grid block representations form a fine-scale grid block feature sequence. A fine-level grid block representation embedding is obtained by adding a global representation and a learnable positional embedding to the fine-scale grid block feature sequence.

3. The method for intelligent product recognition in unmanned vending machines from coarse to fine according to claim 2, characterized in that, The coarse-level grid block representation is input into the Transformer encoder to obtain coarse-level global features, including: A query vector is generated based on the global representation, and a key matrix and a value matrix are generated based on the coarse-scale grid block feature sequence. The query vector and the key matrix are interacted to calculate the attention weight matrix for each coarse-scale grid block; Based on the attention weight matrix and the value matrix, a coarse-level global feature based on self-attention is generated.

4. The method for intelligent product recognition in unmanned vending machines from coarse to fine according to claim 1, characterized in that, Based on the coarse-scale product region mask, the fine-scale grid blocks within the product region and their corresponding position embeddings are selected from the fine-scale grid blocks, and the fine-level representation embedding of the product region is calculated using the following formula: in, To select the precise-scaled grid blocks located within the product area mask, For coarse-scale commodity area masking, Embedding of high-level grid block representations, Embed at the corresponding position of the grid block in the product area. Position embeddings are added to the feature sequences of fine-scale grid blocks. For high-level representation embedding, This is a global representation.

5. The method for intelligent product recognition in unmanned vending machines from coarse to fine according to claim 4, characterized in that, Based on the refined hierarchical representation embedding, refined hierarchical product features are obtained, including: The refined hierarchical representation is embedded into the Transformer encoder to construct a self-attention-based refined hierarchical product feature, expressed by the following formula: in, As a characteristic of high-end products, This represents the query vector generated by the high-level global representation. and These represent the key matrix and value matrix, respectively, obtained by embedding the refined hierarchical representation of the commodity region through a linear projection layer. Here, D represents the feature dimension of the query vector, which is the activation function.

6. The method for intelligent product recognition in unmanned vending machines from coarse to fine according to claim 1, characterized in that, The coarse-scale grid block representation is fused with the fine-level product features to obtain the fused feature, which is expressed by the following formula: in, As a feature of fusion, This represents the query vector generated by the high-level global representation. and These represent the key matrix and value matrix, respectively, generated by the linear projection layer from the coarse-scale grid block representation.

7. A coarse-to-precise intelligent product recognition device for vending machines, wherein the coarse-to-precise intelligent product recognition device for vending machines is used to implement the coarse-to-precise intelligent product recognition method for vending machines as described in claim 1, characterized in that, include: The acquisition module is used to acquire product images; The grid division and representation generation module is used to divide the commodity image into coarse-scale grid blocks and fine-scale grid blocks, generate coarse-level grid block representations and coarse-level grid block representation embeddings based on the coarse-scale grid blocks, and generate fine-level grid block representations and fine-level grid block representation embeddings based on the fine-scale grid blocks. The coarse-level feature extraction and mask generation module is used to input the coarse-level grid block representation into the Transformer encoder to obtain coarse-level global features, and adaptively obtain a coarse-scale commodity region mask on the coarse-scale grid block. The fine-scale feature extraction module is used to filter out the fine-scale grid blocks in the product region and their corresponding position embeddings from the fine-scale grid blocks based on the coarse-scale product region mask, calculate the fine-scale representation embedding of the product region, and obtain the fine-scale product features based on the fine-scale representation embedding. The fusion module is used to downsample the coarse-scale grid block sequence to obtain a coarse-scale grid block representation, and then fuse the coarse-scale grid block representation with the fine-level commodity features to obtain a fused feature. The classification module is used to input the fused features into the classifier to obtain the product classification result.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted to be loaded by a processor to execute the coarse-to-fine intelligent identification method for unmanned vending machines according to any one of claims 1 to 6.

9. An electronic device, characterized in that, The device includes a processor and a memory, the processor being electrically connected to the memory, the memory being used to store instructions and data, and the processor being used to execute the steps of the coarse-to-fine intelligent identification method for unmanned vending machines as described in any one of claims 1 to 6.

Images