Commodity identification method and system based on double-mask collaborative self-supervised learning framework

Abstract

The embodiment of the invention provides a commodity recognition method based on a double-mask collaborative self-supervised learning framework, and belongs to the technical field of image recognition. The commodity identification method comprises the steps that a sample set of commodity images is adopted to carry out multi-dimensional loss pre-training on a model to obtain a pre-training weight, and the model comprises a T-swin backbone network used for carrying out multi-scale feature extraction; the text primitive self-supervision branch is used for generating text primitives according to the extracted features; the bar code prior alignment branch is used for determining bar code features according to the extracted features; the visual reconstruction branch is used for determining the category of the commodity according to the text primitive and the barcode feature; performing hierarchical fine tuning on the model based on the pre-training weight; and carrying out commodity identification by adopting the model subjected to hierarchical fine adjustment. According to the commodity identification method and system, the identification rate is relatively high under the condition of dealing with the commodities with complex commodity packages.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and more specifically to a product recognition method and system based on a dual-mask collaborative self-supervised learning framework. Background Technology

[0002] In today's retail and automated stores, automatic product recognition technology has become one of the core technologies for smart shelves, unmanned retail, and merchandise management. With the continuous development of image recognition, computer vision, and deep learning technologies, product recognition has evolved from simple image classification to more complex fine-grained recognition, cross-view recognition, and long-tail product recognition. Existing product recognition systems largely rely on machine learning and deep learning models, especially convolutional neural networks (CNNs) and visual transformers (ViTs), to classify and locate products. However, for complex product packaging, current image recognition models struggle to achieve high recognition rates. Summary of the Invention

[0003] The purpose of this invention is to provide a product recognition method and system based on a dual-mask collaborative self-supervised learning framework. This product recognition method and system can achieve a high recognition rate when dealing with products with complex packaging.

[0004] To achieve the above objectives, embodiments of the present invention provide a product recognition method based on a dual-mask collaborative self-supervised learning framework, comprising: The model is pre-trained using a sample set of product images to obtain pre-trained weights through multi-dimensional loss. The model includes: The T-swin backbone network is used for multi-scale feature extraction. The text primitive self-supervised branch is used to generate text primitives based on the extracted features; The barcode prior alignment branch is used to determine barcode features based on the extracted features. The visual reconstruction branch is used to determine the category of goods based on text primitives and barcode features; The model is fine-tuned hierarchically based on pre-trained weights; Product recognition is performed using a model that has undergone hierarchical fine-tuning.

[0005] Optionally, the text primitive self-supervised branch is used for: The Textness Head unit is used to generate a textual probability map based on the extracted features; The textual probability graph is encoded by using VQ-VAE primitive discretization units; The masked primitive recovery unit generates the corresponding text primitive based on the encoding.

[0006] Optionally, the encoding of the textual probability graph is mapped using VQ-VAE primitive discretization units, including: Select the text block with the highest preset percentage of probability from the text probability graph; The extracted text blocks are patched to obtain feature representations; The feature representation is input into the encoder to obtain a continuous feature vector; The VQ method is used to map continuous feature vectors to the nearest discrete representation in the codebook to obtain the encoding.

[0007] Optionally, the barcode prior alignment branch is used for: A spatial attention computation unit is used to generate a spatial attention map based on the extracted features; Perform barcode probability prediction on the spatial attention map; The results of the barcode probability prediction operation are binarized to obtain a barcode probability map; Determine the orientation field of the barcode based on the barcode probability map; Extract the stripe sequence based on the barcode's orientation field and barcode probability map; The stripe sequence is subjected to feature reconstruction to obtain the barcode.

[0008] Optionally, the visual reconstruction branch is used for: A binary mask is generated based on the extracted features to represent the masked region; A masking map is generated based on the binary mask and the image; The masking image, text primitives, and barcode features are input into the decoder to identify the product.

[0009] Optionally, a binary mask is generated based on the extracted features to represent the masked region, including: In the first preset stage, a block of a preset pixel size is randomly selected as the mask; In the pre-defined second stage, a new mask is generated on the mask generated in the first stage using the difficult example masking method. In the preset third stage, a preset percentage of masking blocks are randomly sampled from the mask generated in the second stage and used as the first mask block; The unsampled masking block is randomly sampled using the barcode probability map of BPA to obtain the second masking block; Use the first mask block and the second mask block as the new mask.

[0010] Optionally, the model is fine-tuned hierarchically based on the pre-trained weights, including: Extract the parameters from the T-swin backbone network and remove the text primitive self-supervised branch, barcode prior alignment branch, and visual reconstruction branch; Construct a task adapter head, wherein the task adapter head includes a normalization layer, a linear projection layer and a softmax layer connected sequentially after the T-swin backbone network, wherein the learning rate of the softmax layer is greater than the learning rate of the linear projection layer, and the learning rate of the linear projection layer is greater than the learning rate of the normalization layer. The cross-entropy loss function is used as the fine-tuning loss function for the task adapter head.

[0011] On the other hand, the present invention also provides a product recognition system based on a dual-mask collaborative self-supervised learning framework, the system including a processor configured to perform any of the methods described above.

[0012] In another aspect, the present invention also provides an electronic device comprising a processor and a memory, the memory storing instructions, the processor being configured to read the instructions to perform any of the methods described above.

[0013] In another aspect, the present invention also provides a computer-readable storage medium storing instructions for being read by a machine to cause the machine to perform any of the methods described above.

[0014] Through the above technical solution, the embodiments of the present invention provide a product recognition method and system based on a dual-mask collaborative self-supervised learning framework. The product recognition method and system pre-trains a multi-dimensional loss model, and then performs hierarchical fine-tuning of the main model based on the pre-training results. Through a two-step training method, the product features are extracted and recognized in multiple dimensions, thereby achieving effective recognition of products with complex packaging.

[0015] Other features and advantages of the embodiments of the present invention will be described in detail in the following detailed description section. Attached Figure Description

[0016] The accompanying drawings are provided to further illustrate embodiments of the present invention and form part of the specification. They are used together with the following detailed description to explain the embodiments of the present invention, but do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of a product recognition method based on a dual-mask collaborative self-supervised learning framework according to an embodiment of the present invention; Figure 2 This is a flowchart of a method for obtaining a training sample set according to an embodiment of the present invention; Figure 3This is a schematic diagram of a model structure according to one embodiment of the present invention; Figure 4 This is a flowchart of a method for generating text primitives according to an embodiment of the present invention; Figure 5 This is a flowchart of a method for barcode prior alignment branch execution according to an embodiment of the present invention; Figure 6 This is a flowchart of a method for executing a visual reconstruction branch according to an embodiment of the present invention; Figure 7 This is a flowchart of a layered fine-tuning method according to an embodiment of the present invention. Detailed Implementation

[0017] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the scope of the present invention.

[0018] It should be noted that the acquisition, transmission, storage, use, and processing of data in the technical solution of this application all comply with relevant laws and regulations. In the embodiments of this application, certain existing industry solutions such as software, components, and models may be mentioned. These should be considered exemplary, intended only to illustrate the feasibility of implementing the technical solution of this application, and do not imply that the applicant has already used or necessarily used such solutions.

[0019] like Figure 1 The diagram shows a flowchart of a product recognition method based on a dual-mask collaborative self-supervised learning framework according to an embodiment of the present invention. Figure 1 The product identification method may include the following steps: In step S10, a model training sample set is constructed based on the product images; In step S11, the model is pre-trained based on multi-dimensional loss to obtain pre-trained weights; In step S12, the model is fine-tuned hierarchically based on the pre-trained weights; In step S13, the product identification is performed using the layered fine-tuned model.

[0020] Based on this Figure 1 The product recognition method shown in the figure pre-trains the multi-dimensional loss model in step S11, and then fine-tunes the main model in layers based on the pre-training results. Through a two-step training method, the product features are extracted and recognized in multiple dimensions, thus achieving effective recognition of products with complex packaging.

[0021] In such Figure 1In the illustrated product recognition method, step S10 can be used to construct a model training sample set based on the product image. The specific method for obtaining this training sample set can be of various forms known to those skilled in the art. In one example of the present invention, considering the unique characteristics of color and exposure in images of complexly packaged products, the method for obtaining the training sample set may include, for example... Figure 2 The steps shown. Specifically, in this Figure 2 In this context, methods for obtaining a training sample set may include the following steps: In step S20, an unlabeled image dataset is obtained; In step S21, each image in the unlabeled image dataset undergoes two data augmentation operations sequentially. Each data augmentation operation may begin with a random cropping operation on the image. The size of this random cropping operation can be any value known to those skilled in the art. In one example of the present invention, the size of this random cropping operation can be 224. 224. Then, a random horizontal flip operation is performed on the image after the random cropping operation. In one example of the present invention, the probability value of this random horizontal flip operation can be 0.5. Next, a color dithering operation is performed on the image after the random horizontal flip operation. In one example of the present invention, the parameter values ​​of this color dithering operation can be, for example: brightness=0.4, contrast=0.4, saturation=0.4, hue=0.1. Then, a random grayscale operation is performed on the image after the color dithering operation. In one example of the present invention, the probability value of this random grayscale operation can be 0.2. Finally, a Gaussian blur operation is performed on the image after the random grayscale operation. In one example of the present invention, the probability value of the Gaussian blur operation is... The value can be, for example, [0.1, 2.0].

[0022] In step S22, the images after the two data augmentation operations are labeled to obtain the training sample set. Specifically, in one example of the present invention, the labeling operation may be to assign a label to each image, thereby facilitating the comparison of the expected output and the actual output during training.

[0023] Step S11 can be used to pre-train the model based on multi-dimensional loss to obtain pre-trained weights. In one example of the present invention, the model may include, for example: Figure 3 The structure shown. Specifically, in this Figure 3 The model can include a T-Swin backbone network 1, a text primitive self-supervised branch 2, a barcode prior alignment branch 3, and a visual reconstruction branch 4. The T-Swin backbone network 1 can be used for multi-scale feature extraction. Specifically, the T-Swin backbone network 1 can convert the original image into a tensor representation. Its shape is [B,3,H,W], where B represents the batch size, and H and W represent the height and width of the image, respectively.

[0024] The Text Primitive Self-supervision (TIM) branch 2 can be used to generate text primitives based on extracted features. These text primitives can represent feature representations obtained by further feature processing on the feature map (extracted features) of the corresponding dimension. The specific method for generating these text primitives can be of various forms known to those skilled in the art. In one example of this invention, the method for generating text primitives may include, for example... Figure 4 The steps shown. Specifically, in this Figure 4 In this context, methods for generating text primitives may include the following steps: In step S30, a Textness Head unit is used to generate a textual probability map based on the extracted features. Specifically, in this example, the Textness Head unit can be used to detect textual regions in an image and calculate the edge features of the image using the Sobel operator. And generate the text probability map. More specifically, the Textness Head unit can be a loss function for the text probability map using the following formulas (1) to (4): (1) in, Let the edge consistency loss function be... For the generated textual probability graph, For edge features extracted using the Sobel operator, This is the normalization operation function. Let L1 function be used. Formula (1) can be used to represent edge consistency requirements, mainly because the edges of text areas on product packaging are generally more obvious (e.g., the stroke edges of text), while the background is relatively smooth.

[0025] (2) in, is the sparsity loss function. Formula (2) can be used to determine the sparsity loss, which takes into account that the text area on the product packaging is relatively small and needs to be used to avoid misjudging non-text areas as text.

[0026] (3) in, For cross-enhanced consistency functions, , This is a textual probability map after different enhancement operations are performed on the same image. Formula (3) can be used to represent cross-enhancement consistency loss, which is to ensure that the text regions are detected in the processed image after different enhancement operations (such as brightness adjustment, slight cropping, etc.) are performed on the same image.

[0027] (4) in, The loss function for the textual probabilistic graph is... , and The corresponding weights.

[0028] In step S31, the textual probability map is encoded using VQ-VAE (Vector Quantized Variational Autoencoder) primitive discretization units. Specifically, step S31 may involve first selecting the text blocks with the highest probability at a preset percentage (e.g., 20%) from the textual probability map, and then performing patch processing on the extracted text blocks (each patch can have a shape of 16). 16), to obtain the corresponding feature representation; then input the feature representation into the encoder to obtain a continuous feature vector. The continuous feature vector can be obtained by extracting the output of the encoder using a convolutional neural network (CNN); finally, the VQ method is used to map the continuous feature vector to the nearest discrete representation in the codebook to obtain the corresponding token (i.e., the code). The loss function of the VQ-VAE primitive discretization unit can be expressed by the following formula (5): (5) in, The loss function of the VQ-VAE primitive discretization unit. The expected token to be output. This is the actual output token. As a balancing term, For continuous eigenvectors, For text blocks Discrete representation of the mapping, This represents the L2 function.

[0029] In step S32, a masked primitive recovery unit is used to generate the corresponding text primitive based on the encoding. Specifically, the method for generating the text primitive can be through a preset sub-model. The training method for this sub-model can be to first randomly mask tokens in the text region, and then train the sub-model to predict the masked tokens based on the context. More specifically, the training method can include the following steps: First, a token sequence is formed based on the encoding, that is: each text block is... , ,…, Arrange them in spatial order to obtain the token sequence.

[0030] Then, the resulting token sequence is randomly masked. In this example, 15% to 30% of the positions can be randomly selected for masking, replacing them with the [MASK] token to facilitate prediction by the sub-model. An example of the randomly masked token sequence is shown below: .

[0031] The sub-model is then used to predict the masked token. Specifically, the masked sequence can be input into the TokenHead module, which predicts the token at the masked position. In particular, the model needs to predict the original token at each [MASK] position based on the context information.

[0032] Finally, the prediction loss is calculated, and its loss function can be expressed as the following formulas (6) and (7): (6) in, Let cross-entropy be the loss function. The cross-entropy loss function is... For the predicted token, A real token; (7) in, The loss function for the mask primitive recovery unit, , and The corresponding weights.

[0033] Barcode prior alignment branch 3 can be used to perform, for example Figure 5 The method shown. Specifically, in this Figure 5 In this context, the barcode prior alignment branch 3 can be used to perform the following steps: In step S40, a spatial attention computation unit is used to generate a spatial attention map based on the extracted features. Specifically, step S40 can generate the spatial attention map using formulas (8) and (9): (8) (9) in, (·→·)(·) is a two-dimensional convolution mapping operator. The number of input feature channels, For the input feature map, This is a spatial attention map. It is the sigmoid function; In step S41, a barcode probability prediction operation is performed on the spatial attention map. Specifically, step S41 can be achieved by using formulas (10) and (11) to determine the result of the barcode prediction probability operation: (10) (11) in, This is the result of the barcode prediction probability operation. This is element-wise multiplication (weighted according to spatial location), which means using spatial attention to weight the feature map to highlight the barcode region features.

[0034] In step S42, the result of the barcode probability prediction operation is binarized to obtain a barcode probability map.

[0035] In step S43, the orientation field of the barcode is determined based on the barcode probability map. Specifically, step S43 may first use formulas (12) and (13) to determine the gradient characteristics of the orientation field: (12) (13) in, The gradient feature is horizontal. The size parameter of the convolution kernel. Padded parameters are used to fill the boundaries of the convolution operation. It is a vertical gradient feature; Then, the direction vector is determined using formulas (14) and (15): (14) Formula (14) can be used to merge horizontal and vertical gradient information to obtain the direction vector of each pixel. (15) in, The (·,·) operator performs a concatenation operation, which concatenates two feature maps along the channel dimension to obtain a fused feature map. Here is the direction vector. Formula (15) represents the calculation of the direction vector for each pixel through a convolutional layer; Finally, the direction field is determined using formulas (16) to (18). (16) (17) (18) in, To represent summation on the second and third dimensions of the tensor (i.e., spatial dimensions H and W), and to achieve aggregation operations over the entire spatial region, It is a very small positive number (stable term / smoothing term) used to avoid numerical instability caused by a denominator of 0 or too small (such as division by zero error or gradient explosion). For L2 normalization function, For the direction field, =2 indicates that L2 norm normalization is used.

[0036] In step S44, the stripe sequence is extracted based on the barcode orientation field and the barcode probability map. Specifically, in step S34, the barcode can be rotated first using formulas (19) to (21): (19) (20) ,(twenty one) in, The angle of rotation, The arctangent function (two-parameter arctangent) is used to calculate the direction angle of a vector based on its two components. This is an affine sampling grid generation function, used to generate a sampling coordinate grid based on a given affine transformation matrix, so as to realize spatial transformation (such as rotation, translation, scaling) of the input feature map. Let be a rotation matrix. Input feature map Size information (including batch, number of channels, and spatial dimensions). The result of barcode rotation. This is a grid sampling function used to sample the input feature map based on the grid. Interpolation sampling is performed to obtain the output feature map after spatial transformation; Then, formulas (22) and (23) are used to cut the barcode area. ,(twenty two) ,(twenty three) in, The result of cropping. This is a bounding box calculation function used to determine the minimum bounding rectangle based on the regions of 1 in the binary mask, thus obtaining the spatial range of the barcode region. For binarization mask, For clipping and size normalization functions, used to extract from... According to Crops out the barcode ROI from a specified area and scales / resamples the cropped result to a preset output size. This represents the length of the barcode area in the horizontal direction (stripe sequence length). Finally, formulas (24) to (27) are used to determine the stripe sequence based on the cutting results: ,(twenty four) (25) (26) (27) in, This is an adaptive two-dimensional average pooling operator used to average and aggregate the input feature maps in the spatial dimension, and adjust the spatial size of the output feature map to the target size [1,L]. The output height is 1 and the output width is L. It is a stripe sequence. This is a rounding function used to round continuous values ​​to the nearest integer. Mapped to discrete values, This is a gradient truncation / stop operation used to block the gradient propagation of terms within parentheses during backpropagation, ensuring that backpropagation still proceeds with continuous values. The gradient is the primary factor, thereby achieving a differentiable binarization strategy of "forward discretization and backward differentiability". In step S45, feature reconstruction is performed on the stripe sequence to obtain the barcode. Specifically, step S15 may involve first performing feature encoding and fusion operations using formulas (28) to (30). (28) (29) (30) in, This is the result of feature encoding and fusion operations. This is an adaptive two-dimensional average pooling operator used to globally average and aggregate the input feature maps across spatial dimensions (H, W), resulting in an output spatial size of 1×1, thus obtaining a global feature vector. The feature map after the barcode region is completed / reconstructed (which can be understood as the feature representation after being processed by the preceding barcode reconstruction or completion module). This is a feature concatenation operation used to concatenate two feature vectors along the channel dimension (feature dimension). Then, formulas (31) to (35) are used to reconstruct the (multilayer perceptron) stripe sequence. (31) (32) (33) (34) (35) in, For the reconstructed stripe sequence; The loss function of the spatial attention calculation unit adopts the following formulas (36) to (41). (36) in, This is the Binary Cross Entropy (BCE) loss function, used to measure the difference between the predicted sequence and the target binary sequence. It is suitable for outputting probabilistic values ​​in the range [0,1]. The binarized result of the stripe sequence (target sequence) is used as a monitoring signal. This is the loss aggregation parameter, used to specify the strategy for summarizing the loss at each location. The average aggregation method is used, which means that the loss of all positions is averaged to obtain the final loss; (37) (38) in, This is the normalized orientation vector map (or orientation field), representing the orientation vector of each pixel. The average direction vector (local main direction / reference direction vector) within the barcode area is used as the alignment target for directional consistency. (39) in, This is the standard deviation function, used to calculate the dispersion of the interval sequence. This is the mean function, used to calculate the average spacing of the interval sequence. Batch size; (40) in, , , These represent the barcode probability maps predicted at different scales (high / medium / low resolution); when calculating the loss, the medium and low resolution prediction results can be upsampled to the same spatial size as the high resolution prediction results and then compared; MSE( , ) is the mean squared error function, used to measure the difference between barcode probability maps of different scales; (41) in, The normalized direction vector map at pixel position Local direction unit vector at; This is the global main direction unit vector of the barcode area; For the barcode probability map at pixel position ( The probability weight at () is used to emphasize the barcode area and suppress the background area; (42) in, The loss function for the spatial attention computation unit. , , , , , The corresponding weights.

[0037] Visual reconstruction branch 4 can be used to determine the category of a product based on text primitives and barcode features. Specifically, in one example of the invention, this visual reconstruction branch 4 can be used to perform actions such as... Figure 6 The steps are shown. In this Figure 6 In this process, visual reconstruction branch 4 can be used to perform the following steps: In step S50, a binary mask is generated based on the extracted features to represent the masked region. In one example of the present invention, step S50 may be used for: in a preset first stage, randomly selecting a block of a preset pixel size as a mask; in a preset second stage, generating a new mask on the mask generated in the first stage using a hard-example masking method; in a preset third stage, randomly sampling a preset percentage of masking blocks on the mask generated in the second stage as a first mask block; randomly sampling the unsampled masking blocks using a barcode probability map (BPA) to obtain a second mask block; and using the first and second mask blocks as a new mask. More specifically, in one example of the present invention, step S50 may be used to generate a binary mask. The mask is used to represent the occluded area, where 1 represents the occluded part and 0 represents the unoccluded part. The mask generation adopts a hybrid strategy and schedule to gradually increase the training difficulty.

[0038] In the initial stage (the first phase, such as the first 5–10 epochs), block-based masking is used, which involves randomly selecting blocks of 16–32 pixels in size for masking. =0.6; Difficult example masking is introduced in the middle stage (second phase): =0.3–0.5, =0.65, prioritize masking text or barcode areas, obtained using TIM's Htxt (Textness Heatmap). Select the top-20% region, randomly sample 50% of the masked blocks from it, and randomly sample the remaining 50%; perform the same filtering using BPA's Pbar (barcode probability map), and improve performance later. =0.7–0.75, and increase the randomness of the block size (from 16–32 pixels to 12–36).

[0039] In step S51, a masking image is generated based on the binary mask and the image. In one example of the present invention, the generated masking image may be as shown in the following formula (47): (47) in, For the generated masking map, For the pixel values ​​to fill (using the mean color: = x.mean(dim=(2,3), keepdim=True)).

[0040] In step S52, the masking image, text primitives, and barcode features are input into the decoder to identify the product. Specifically, in this example, step S52 can be determined by calculating the reconstruction loss function of the decoder according to formulas (43) to (46): (43) in, To reconstruct the loss function, For pixel reconstruction loss, For frequency domain reconstruction loss, For edge consistency loss, , and For the corresponding weights, in one example, =1.0, =0.5, = ; (44) in, For the desired output, For the actual output, For pixel multiplication, For mask; (45) in, It is the Fourier transform function; (46) in, To extract the expected edge features using the Sobel operator, The actual edge features are extracted using the Sobel operator.

[0041] Step S12 can be used to perform hierarchical fine-tuning of the model based on pre-trained weights. The specific method for this hierarchical fine-tuning can be of various forms known to those skilled in the art. In one example of the present invention, considering the already constructed model, the hierarchical fine-tuning method may include, for example... Figure 7 The steps are shown. In this Figure 7 In this context, the method of layered fine-tuning may include the following steps: In step S60, the parameters in the T-swin backbone network are extracted, and the text primitive self-supervised branch, barcode prior alignment branch, and visual reconstruction branch are removed. In step S61, a task adapter head is constructed, wherein the task adapter head includes a normalization layer, a linear projection layer and a softmax layer connected sequentially after the T-swin backbone network. The learning rate of the softmax layer is greater than the learning rate of the linear projection layer, and the learning rate of the linear projection layer is greater than the learning rate of the normalization layer. In step S62, the cross-entropy loss function is used as the fine-tuning loss function for the task adapter head.

[0042] Furthermore, regarding the deployment process of this product identification method in application, in one example of this invention, it can be deployed based on an end-to-end inference method. Specifically, the deployment process may include the following steps: Step 1: Image Acquisition and Preprocessing Receive raw video stream frames from the camera or trigger still images to be captured, and normalize the image pixels using the same mean and variance as during the training phase.

[0043] Step 2, Feature Forward Propagation: The preprocessed tensors are input into the fine-tuned model. The backbone network utilizes the text sensitivity (given by the TIM branch) and barcode structure sensitivity (given by the BPA branch) learned during the pre-training phase to extract high-dimensional feature vectors containing rich semantics. It is worth noting that even if the text in the image is blurred or the barcode is partially occluded, the model can still effectively complete the image using contextual features.

[0044] Step 3: Result Post-processing and Confidence Filtering Obtain the probability vector P = [p1, p2, ..., pC] output by the task header. Select the K candidate categories with the highest probabilities. Set a confidence threshold. (For example, 0.85). If If the identification is successful, the corresponding product ID and name will be output. If the product is identified as "unknown product" or "low confidence level", the system will prompt the user to "adjust the product angle" or "manual intervention".

[0045] Step 4: Model Lightweighting and Edge Deployment To meet the requirements of real-time performance (low latency) and low computing power in unmanned retail scenarios, the following optimizations were implemented: (1) Model format conversion: Export the weights of the .pth model generated by the training framework (such as PyTorch) as the intermediate representation format ONNX (OpenNeural Network Exchange), eliminating the dependency on specific deep learning frameworks.

[0046] (2) Operator fusion and precision quantization: Use an inference engine (such as NVIDIA TensorRT or OpenVINO). Combine convolutional layers, batch normalization (BN) layers, and activation function layers into a single computation kernel to reduce memory accesses. While maintaining an accuracy loss of less than 1%, compress the model parameters from 32-bit floating-point numbers (FP32) to 8-bit integers (INT8). This can reduce the model size by 4 times and improve inference speed by 3-5 times.

[0047] (3) Service encapsulation: The optimized model is encapsulated as a gRPC or RESTful API service. Input is Base64 encoded image data. Output is a JSON format result (containing: product ID, product name, confidence level, and processing time).

[0048] On the other hand, the present invention also provides a product recognition system based on a dual-mask collaborative self-supervised learning framework, the system including a processor configured to perform any of the methods described above.

[0049] In another aspect, the present invention also provides an electronic device comprising a processor and a memory, the memory storing instructions, the processor being configured to read the instructions to perform any of the methods described above.

[0050] In another aspect, the present invention also provides a computer-readable storage medium storing instructions for being read by a machine to cause the machine to perform any of the methods described above.

[0051] Through the above technical solution, the embodiments of the present invention provide a product recognition method and system based on a dual-mask collaborative self-supervised learning framework. The product recognition method and system pre-trains a multi-dimensional loss model, and then performs hierarchical fine-tuning of the main model based on the pre-training results. Through a two-step training method, the product features are extracted and recognized in multiple dimensions, thereby achieving effective recognition of products with complex packaging.

[0052] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0053] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0054] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0055] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0056] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0057] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0058] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0059] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0060] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A product recognition method based on a dual-mask collaborative self-supervised learning framework, characterized in that, include: The model is pre-trained using a sample set of product images to obtain pre-trained weights through multi-dimensional loss. The model includes: The T-swin backbone network is used for multi-scale feature extraction. The text primitive self-supervised branch is used to generate text primitives based on the extracted features; The barcode prior alignment branch is used to determine barcode features based on the extracted features. The visual reconstruction branch is used to determine the category of goods based on text primitives and barcode features; The model is fine-tuned hierarchically based on pre-trained weights; Product recognition is performed using a model that has undergone hierarchical fine-tuning.

2. The method according to claim 1, characterized in that, The self-supervised branch of the text primitives is used for: The Textness Head unit is used to generate a textual probability map based on the extracted features; The textual probability graph is encoded by using VQ-VAE primitive discretization units; The masked primitive recovery unit generates the corresponding text primitive based on the encoding.

3. The method according to claim 2, characterized in that, The encoding of the textual probability graph is mapped using VQ-VAE primitive discretization units, including: Select the text block with the highest preset percentage of probability from the text probability graph; The extracted text blocks are patched to obtain feature representations; The feature representation is input into the encoder to obtain a continuous feature vector; The VQ method is used to map continuous feature vectors to the nearest discrete representation in the codebook to obtain the encoding.

4. The method according to claim 1, characterized in that, The barcode prior alignment branch is used for: A spatial attention computation unit is used to generate a spatial attention map based on the extracted features; Perform barcode probability prediction on the spatial attention map; The results of the barcode probability prediction operation are binarized to obtain a barcode probability map; Determine the orientation field of the barcode based on the barcode probability map; Extract the stripe sequence based on the barcode's orientation field and barcode probability map; The stripe sequence is subjected to feature reconstruction to obtain the barcode.

5. The method according to claim 2, characterized in that, The visual reconstruction branch is used for: A binary mask is generated based on the extracted features to represent the masked region; A masking map is generated based on the binary mask and the image; The masking image, text primitives, and barcode features are input into the decoder to identify the product.

6. The method according to claim 5, characterized in that, A binary mask is generated based on the extracted features to represent the masked region, including: In the first preset stage, a block of a preset pixel size is randomly selected as the mask; In the pre-defined second stage, a new mask is generated on the mask generated in the first stage using the difficult example masking method. In the preset third stage, a preset percentage of masking blocks are randomly sampled from the mask generated in the second stage and used as the first mask block; The unsampled masking block is randomly sampled using the barcode probability map of BPA to obtain the second masking block; Use the first mask block and the second mask block as the new mask.

7. The method according to claim 1, characterized in that, Based on the pre-trained weights, the model is fine-tuned hierarchically, including: Extract the parameters from the T-swin backbone network and remove the text primitive self-supervised branch, barcode prior alignment branch, and visual reconstruction branch; Construct a task adapter head, wherein the task adapter head includes a normalization layer, a linear projection layer and a softmax layer connected sequentially after the T-swin backbone network, wherein the learning rate of the softmax layer is greater than the learning rate of the linear projection layer, and the learning rate of the linear projection layer is greater than the learning rate of the normalization layer. The cross-entropy loss function is used as the fine-tuning loss function for the task adapter head.

8. A product recognition system based on a dual-mask collaborative self-supervised learning framework, characterized in that, The system includes a processor configured to perform the method as described in any one of claims 1 to 7.

9. An electronic device, characterized in that, The electronic device includes a processor and a memory, the memory storing instructions, and the processor being configured to read the instructions to perform the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that are read by a machine to cause the machine to perform the method as described in any one of claims 1 to 7.

Images