Multi-target commodity recognition method, device and system based on multimodal data processing

Abstract

The present invention relates to the field of intelligent vending technology, solves the problem in the prior art that it is impossible to accurately identify goods in multi-target scenarios, and provides a multi-target commodity identification method, device, and system based on multimodal data processing. The method includes: acquiring multiple frames of real-time images in a commodity transaction scenario; pre-processing and extracting label information from the real-time images to determine the text information corresponding to the target image and the commodity label; performing instance segmentation on the target image to determine the commodity location information; performing feature extraction on the target image to determine the commodity image feature information; fine-tuning and optimizing the open source multimodal visual language model based on pre-collected multi-source privatized data in the intelligent vending scenario to obtain a multimodal large model; inputting the commodity image feature information and text information into the multimodal large model for information fusion to determine the commodity target recognition result. The present invention can accurately identify commodities in multi-target scenarios.

Description

Technical Field

[0001] The present invention relates to the field of intelligent vending technology, and in particular to a multi-target commodity recognition method, device and system based on multimodal data processing. Background Art

[0002] Smart vending machines, a common type of unmanned vending machine, are widely used in various fields, such as shopping, food, beverages, and daily necessities. Traditional smart vending machines primarily rely on barcode or QR code scanning to identify products. This approach often relies on manual input of product information and label matching. However, this method has limitations, such as the inability to effectively identify products that are occluded, overlapped, or have changed position. Furthermore, manually annotated label information is prone to errors, especially when products are frequently updated, making it difficult for vending machines to promptly adapt to the display and identification of new products. Therefore, how to efficiently and accurately identify products in dynamic environments, especially in complex scenarios such as the simultaneous appearance, occlusion, and overlap of multiple product targets, has become a core issue in the intelligent upgrade of smart vending machines.

[0003] Existing technologies typically rely on single visual or textual information for multi-product recognition, making them incapable of handling the complexities of dynamic transaction scenarios. For example, traditional image recognition methods are prone to misidentification or omission when dealing with occluded or overlapping items. Furthermore, text extraction also faces challenges with unclear, ambiguous, or positionally shifted labels. Furthermore, existing methods often rely on single-modality processing, lacking comprehensive analysis of visual, spatial, and semantic information. Consequently, existing technologies cannot meet the real-time, accurate product recognition requirements of smart vending machines in complex scenarios.

[0004] Existing Chinese patent CN114445201A discloses a combined product retrieval method and system based on a multimodal pre-trained model, comprising: dividing product images into single product images and combined product images; training a combined product image detector; acquiring and combining the feature encoding, position encoding, and segment encoding of the text modality and image module in the combined product image to learn an embedded representation, which is then input into the constructed multimodal pre-trained model; using the multimodal pre-trained model to extract retrieval features of the image modality and text modality of the single product image; the multimodal pre-trained model extracts retrieval features of the combined product image image and text fusion based on the bounding box and bounding box features of each target product in the combined product image, calculates the pre-distance between the combined product features and the single product features in the search library as the product similarity, and selects the most similar single product as the result and returns it. The above patent solution cannot accurately handle situations such as occlusion and overlap between products, and cannot cope with the differences between different product features and text descriptions. Therefore, it is difficult to ensure accurate product recognition in real-world scenarios.

[0005] Therefore, how to accurately identify products in multi-target scenarios is an urgent problem to be solved. Summary of the Invention

[0006] In view of this, the present invention provides a multi-target commodity recognition method, device and system based on multimodal data processing to solve the problem in the prior art that commodities cannot be accurately recognized in multi-target scenarios.

[0007] The technical solution adopted in the present invention is:

[0008] In a first aspect, the present invention provides a multi-target commodity identification method based on multimodal data processing, the method comprising:

[0009] Acquire real-time video data in a commodity trading scenario, and decompose the real-time video data into multiple frames of real-time images;

[0010] Preprocessing and label information extraction are performed on the real-time image to determine text information corresponding to the preprocessed target image and the product label;

[0011] Performing instance segmentation on the target image to determine product location information;

[0012] Extracting features of the target image based on the product location information to determine product image feature information;

[0013] Based on pre-collected multi-source private data from smart vending scenarios, we fine-tune and optimize the open-source multimodal visual language model to obtain a large multimodal model for product recognition.

[0014] The commodity image feature information and the text information are input into the multimodal large model for information fusion, and the commodity target recognition result is determined based on the fused feature information.

[0015] Preferably, the preprocessing of the real-time image and the extraction of label information, and determining text information corresponding to the preprocessed target image and the product label includes:

[0016] Performing size adjustment and noise reduction processing on the real-time image to determine the target image;

[0017] Performing target detection on the target image to determine product area location information;

[0018] According to the commodity area location information, commodity labels in the commodity area are processed using optical character recognition technology to determine the text information.

[0019] Preferably, performing instance segmentation on the target image to determine product location information includes:

[0020] Extracting feature information of the product area through a convolutional neural network based on the product area location information, and determining a candidate area based on the extracted feature information;

[0021] Processing the candidate regions through an instance segmentation network to determine a binary image corresponding to each commodity target;

[0022] The binary image is processed using post-processing technology to determine the commodity location information.

[0023] Preferably, the method of fine-tuning and optimizing the open source multimodal visual language model based on pre-collected multi-source private data in the smart vending scenario to obtain a large multimodal model for commodity recognition includes:

[0024] Based on the pre-collected multi-source raw data in the smart vending scenario, the multi-source raw data is cleaned and structured to obtain a labeled data set;

[0025] Based on the image information and text labels in the annotated dataset, the image and text data are paired and constructed, and the format is converted and uniformly preprocessed to obtain a multimodal input sample set for training;

[0026] Based on the multimodal input sample set, we load weight parameters into the open-source visual language pre-training model and build a visual encoding and language encoding network structure that supports joint optimization to obtain the initial structure of the multimodal model for fine-tuning.

[0027] Fine-tuning the initial structure of the multimodal model according to the characteristics of the vending scenario and the recognition accuracy requirements, and optimizing the hyperparameter configuration through a cross-validation strategy to obtain multiple candidate multimodal models;

[0028] According to the preset accuracy, recall rate and response time, each candidate multimodal model is evaluated and compared to obtain the multimodal large model.

[0029] Preferably, the step of inputting the product image feature information and the text information into the multimodal large model for information fusion, and determining the product target recognition result based on the fused feature information includes:

[0030] Inputting the product image feature information and the text information into the multimodal large model to obtain fused feature information that fuses image features and text semantics;

[0031] Inputting the fused feature information into a pre-trained product classification model to obtain an initial product category;

[0032] According to the initial product category, determining whether there are similar products in the current initial product category;

[0033] When similar products exist, obtaining a local area of ​​features to be extracted and a target feature to be extracted according to the initial product category;

[0034] Extracting features of the target image based on the local region and the target features to obtain local region feature information;

[0035] The initial commodity category is classified according to the local area feature information to obtain a target commodity category as the commodity target recognition result.

[0036] Preferably, when similar products exist, obtaining the local area of ​​the feature to be extracted and the target feature to be extracted according to the initial product category includes:

[0037] According to the initial product category, sample images corresponding to multiple subcategories under the category are selected from a preset product image database;

[0038] Inputting each of the sample images into a pre-trained saliency detection model to obtain a saliency heat map, wherein the saliency heat map is used to characterize the area in the sample image where attention to visual features is most focused;

[0039] Performing threshold segmentation on the saliency heat map to obtain multiple candidate regions;

[0040] Performing a comprehensive scoring on each of the candidate regions, and screening the local region from each of the candidate regions according to the scoring results;

[0041] Candidate feature extraction and feature evaluation are performed on the local area, and the target feature is obtained by screening from the extracted candidate features based on the feature evaluation result.

[0042] Preferably, the comprehensive scoring of the candidate regions and the local region obtained according to the scoring results include:

[0043] Obtain the saliency value in the saliency heat map corresponding to each candidate region;

[0044] Calculating the average significance value of each candidate region as a significance score value according to each of the significance values;

[0045] According to the initial product category, each of the sample images is input into a pre-trained image classification model to obtain a category activation map;

[0046] Obtaining a response heat map of each sample image under the current initial product category according to the category activation map;

[0047] Counting the pixels at the corresponding position of each candidate region in the response heat map, and calculating the average activation intensity of each candidate region as a category-related score value;

[0048] Performing weighted fusion on the significance score value of each candidate region and the category-related score value to obtain the score result of each candidate region;

[0049] The scoring results are compared with a preset scoring threshold, and according to the comparison results, at least one region is selected from each candidate region as the local region.

[0050] Preferably, the extracting and evaluating candidate features of the local area, and screening the target feature from the extracted candidate features according to the feature evaluation result, comprises:

[0051] According to the initial product category, obtaining a feature extraction strategy corresponding to each candidate feature corresponding to the category;

[0052] Performing multi-path feature extraction on the local area according to each of the feature extraction strategies to obtain a plurality of candidate feature information;

[0053] Performing a distribution consistency analysis on each candidate feature information in each sample image, and obtaining the frequency and position deviation of each candidate feature information in different sample images as a distribution consistency indicator;

[0054] Input each candidate feature information into a pre-trained product recognition model to obtain a recognition result, and obtain a classification confidence corresponding to the recognition result as a classification response strength of each candidate feature information;

[0055] Evaluate each candidate feature information according to the distribution consistency index and the classification response strength to obtain a feature evaluation result;

[0056] According to the feature evaluation result, the target feature is screened out from each candidate feature information.

[0057] In a second aspect, the present invention provides a multi-target commodity recognition device based on multimodal data processing, the device comprising:

[0058] A real-time image acquisition module is used to acquire real-time video data in a commodity trading scenario and decompose the real-time video data into multiple frames of real-time images;

[0059] A preprocessing and label information extraction module, configured to preprocess the real-time image and extract label information, and determine text information corresponding to the preprocessed target image and the product label;

[0060] An instance segmentation module, configured to perform instance segmentation on the target image to determine product location information;

[0061] A feature extraction module is used to extract features of the target image based on the product location information to determine product image feature information;

[0062] The multimodal large model training module is used to fine-tune and optimize the open source multimodal visual language model based on pre-collected multi-source private data in smart vending scenarios, thereby obtaining a multimodal large model for product recognition.

[0063] The commodity recognition module is used to input the commodity image feature information and the text information into the multimodal large model for information fusion, and determine the commodity target recognition result based on the fused feature information.

[0064] In a third aspect, an embodiment of the present invention further provides a multi-target commodity recognition system based on multimodal data processing, comprising: an image acquisition device, at least one processor, at least one memory, and computer program instructions stored in the memory, wherein when the computer program instructions are executed by the processor, the method described above is implemented.

[0065] In summary, the beneficial effects of the present invention are as follows:

[0066] The present invention provides a multi-target commodity recognition method, device and system based on multimodal data processing, comprising: acquiring real-time video data in a commodity transaction scenario, decomposing the real-time video data into multiple frames of real-time images; preprocessing and extracting label information on the real-time images, and determining text information corresponding to the preprocessed target image and the commodity label; performing instance segmentation on the target image to determine commodity location information; performing feature extraction on the target image based on the commodity location information to determine commodity image feature information; fine-tuning and optimizing an open source multimodal visual language model based on pre-collected multi-source privatized data in a smart vending scenario to obtain a multimodal large model for commodity recognition; inputting the commodity image feature information and the text information into the multimodal large model for information fusion, and determining a commodity target recognition result based on the fused feature information. The present invention extracts multiple frames of images from real-time video data, and through preprocessing and label information extraction, preliminarily identifies the products in each frame of the image, extracts the product labels and corresponding text information, then processes the target image through instance segmentation technology, accurately locates the bounding box of each product and solves the occlusion problem between products, and further extracts image features including visual features, spatial features and semantic features using product location information, and performs information fusion through a multimodal large model to combine the product image features with the extracted text information to enhance the recognition capability of complex product targets. Finally, based on the fusion of image and text information, the model can accurately distinguish and identify multiple product targets, and maintain efficient recognition performance even in complex environments where multiple products exist at the same time and are partially overlapped or occluded. This not only improves the recognition accuracy, but also can operate stably in dynamic scenes to meet the needs of multi-target recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work, and these are all within the scope of protection of the present invention.

[0068] Figure 1 Schematic diagram of the overall working process of the multi-target commodity identification method based on multimodal data processing in Example 1 of the present invention;

[0069] Figure 2 Schematic diagram of the process of preprocessing the real-time image and extracting label information in Example 1 of the present invention;

[0070] Figure 3 Schematic diagram of the process of performing instance segmentation on the target image in embodiment 1 of the present invention;

[0071] Figure 4Schematic diagram of the process of extracting features from the target image in Example 1 of the present invention;

[0072] Figure 5 Schematic diagram of the process of determining the commodity target recognition result in Example 1 of the present invention;

[0073] Figure 6 This is a schematic diagram of a similar product in Example 1 of the present invention;

[0074] Figure 7 This is a structural block diagram of a multi-target commodity recognition device based on multimodal data processing in Example 2 of the present invention;

[0075] Figure 8 Schematic diagram of the structure of a multi-target commodity recognition system based on multimodal data processing in Example 3 of the present invention. DETAILED DESCRIPTION

[0076] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. It should be noted that, in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. In the description of the present invention, it should be understood that the orientation or position relationship indicated by the terms "center", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like is based on the orientation or position relationship shown in the drawings, and is only for the convenience of describing the present application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore cannot be understood as limiting the present invention. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further limitations, elements defined by the phrase "comprising..." do not preclude the presence of additional identical elements in the process, method, article, or apparatus comprising the elements. The embodiments of the present invention and the features thereof may be combined with each other if there is no conflict, and all are within the scope of protection of the present invention.

[0077] Example 1

[0078] See Figure 1 Embodiment 1 of the present invention discloses a multi-target commodity recognition method based on multimodal data processing, the method comprising:

[0079] Acquire real-time video data in a commodity trading scenario, and decompose the real-time video data into multiple frames of real-time images;

[0080] Specifically, real-time video data of the transaction scene is first collected in real time using a camera or other video capture device. This real-time video data contains video data of multiple products from different perspectives. Next, the video stream is decomposed into a series of continuous image frames. Each frame serves as input for subsequent object detection and analysis, laying the foundation for subsequent steps such as product recognition, label extraction, instance segmentation, and feature extraction. By effectively decomposing real-time video data into multiple frames, it is possible to accurately capture the visual information of products in dynamic environments, providing efficient data support for subsequent processing.

[0081] Preprocessing and label information extraction are performed on the real-time image to determine text information corresponding to the preprocessed target image and the product label;

[0082] Specifically, each image frame undergoes image enhancement processing, including noise reduction, brightness, and contrast adjustment, to improve image quality and clarity. This process ensures high readability under varying lighting conditions, facilitating subsequent processing. Next, image preprocessing is performed using image processing algorithms such as edge detection and image smoothing to remove unnecessary background noise and highlight the product area, ensuring clear identification of the product. After image preprocessing, optical character recognition (OCR) technology is used to extract text information from product labels, including product name, price, and SKU number. OCR technology identifies text areas in the image and converts the characters on the product labels into machine-readable text. It also effectively addresses issues such as font size, blur, and rotation to ensure accurate text extraction. This process assigns relevant text information to each product object, providing comprehensive input data for subsequent product recognition, object detection, and feature matching.

[0083] In one embodiment, see Figure 2 The preprocessing of the real-time image and the extraction of label information to determine the text information corresponding to the preprocessed target image and the product label includes:

[0084] Performing size adjustment and noise reduction processing on the real-time image to determine the target image;

[0085] Specifically, the live image is scaled according to the preset input size requirements. This resizing ensures consistent image size across different devices and environments, facilitating efficient subsequent processing. Subsequently, the rescaled image undergoes noise reduction, removing noise such as brightness fluctuations and blur. Common noise reduction algorithms, such as Gaussian blur and median filtering, are applied to the rescaled image to enhance product edges and details, improving image quality. After processing these images, the product area is focused, resulting in a clear target image, laying a solid foundation for subsequent object detection and text recognition steps.

[0086] Performing target detection on the target image to determine product area location information;

[0087] Specifically, object detection algorithms, such as YOLO and Faster R-CNN, are used to process preprocessed target images, identifying and locating all product objects within the image. Convolutional Neural Networks (CNNs) and Region Proposal Networks (RPNs) are then used to effectively classify different objects within the image and calibrate product bounding boxes. These bounding boxes accurately determine the location of the product area, providing location information for subsequent product label extraction and feature analysis. Object detection algorithms efficiently identify product objects in real-time images, even in complex product placement scenarios, with effective handling of occlusions and overlaps, paving the way for accurate product identification.

[0088] According to the commodity area location information, commodity labels in the commodity area are processed using optical character recognition technology to determine the text information.

[0089] Specifically, after obtaining the product area location information, the product area image is cropped from the original target image based on the product area location information. Then, OCR (Optical Character Recognition) technology is applied to process the cropped product label area. OCR technology analyzes the character area in the image to identify the text information therein, such as the product name, price, production date and other important information. To ensure the accuracy of text information extraction, OCR technology will optimize and adjust based on factors such as the font, size, rotation angle and blur, and use adaptive text recognition algorithms to handle different label formats and languages. Ultimately, the text information on the product label will be combined with the image features of the product to provide reliable multimodal data support for subsequent product identification and processing.

[0090] Performing instance segmentation on the target image to determine product location information;

[0091] Specifically, an instance segmentation algorithm (such as Mask R-CNN) is applied to the target image. The goal is to accurately segment each independent product object in the image and generate an accurate mask for each object. Unlike traditional object detection, instance segmentation can not only determine the bounding box of the product, but also accurately distinguish the specific outline and shape of the product. Therefore, the algorithm first extracts image features through a convolutional neural network and generates potential target regions through a region proposal network (RPN). Next, a more refined segmentation network is used to classify these regions at the pixel level, and finally a detailed mask is generated for each product object. In this way, instance segmentation can not only accurately locate the bounding box for each product, but also describe the product's shape information in detail, providing more accurate location information for subsequent product feature extraction and positioning. This process is particularly important for product target recognition in complex backgrounds, and can effectively handle product occlusion, overlap, and similar background issues.

[0092] In one embodiment, see Figure 3 , performing instance segmentation on the target image to determine the commodity location information includes:

[0093] Extracting feature information of the product area through a convolutional neural network based on the product area location information, and determining a candidate area based on the extracted feature information;

[0094] Specifically, the product region location information (i.e., bounding boxes in the target image) is used to extract product regions of interest. These regions serve as input to a convolutional neural network (CNN), which extracts features from the image through multiple convolutional layers. CNNs can identify high-level features such as product shape, color, and texture, which are crucial for distinguishing different products. After feature extraction, a region proposal network algorithm analyzes the extracted features to generate multiple candidate regions representing areas that may contain the product. These candidate regions serve as input for further processing in an instance segmentation network.

[0095] Processing the candidate regions through an instance segmentation network to determine a binary image corresponding to each commodity target;

[0096] Specifically, the candidate regions generated above are fed into an instance segmentation network (such as Mask R-CNN). The network analyzes each candidate region one by one and generates a pixel-level binary mask image for each product target. Each pixel value in this binary mask indicates whether the location belongs to the target product region. This pixel-level segmentation not only accurately identifies the product location, but also clearly defines its specific outline and shape, helping to distinguish multiple overlapping or adjacent product targets.

[0097] The binary image is processed using post-processing technology to determine the commodity location information.

[0098] Specifically, some noise or incomplete areas may appear in the binary image generated by instance segmentation. At this time, post-processing techniques (such as morphological operations, contour extraction, region merging, etc.) are used to further optimize the segmentation results. Morphological operations (such as dilation and erosion) remove small noise and fill gaps in the product area to ensure that the product boundaries are more coherent and accurate. Then, the contour extraction algorithm can be used to clarify the boundary shape of each product target, and the position of the product in the image can be accurately located based on the segmented area. Finally, the product location information obtained after post-processing will serve as the basic data for subsequent steps (such as feature extraction and target recognition).

[0099] Extracting features of the target image based on the product location information to determine product image feature information;

[0100] Specifically, based on the product location information extracted in the previous step, the target image is cropped to obtain a local area containing only the product. Next, a convolutional neural network (CNN) is used to extract features from this product area, extracting deep features, including the product's color, texture, shape, edges, and other information. These features are processed through multiple convolutional layers, pooling layers, and activation functions to transform local details in the image into more abstract high-level feature representations. At the same time, in addition to conventional visual features, more advanced network structures (such as ResNet and VGG) can also be used to extract more complex feature information. During the feature extraction process, position encoding information, such as the relative position of the product in the image, can also be incorporated, so that the extracted features not only reflect the product's appearance characteristics but also take into account its spatial position relationship. In this way, the final product image feature information contains unique visual and spatial representations of the product, supporting subsequent product recognition and matching.

[0101] In one embodiment, see Figure 4 , performing feature extraction on the target image based on the product location information to determine product image feature information includes:

[0102] Determining, based on the commodity location information, location information of a to-be-processed area related to the commodity in the target image;

[0103] Specifically, based on the product location information (such as bounding box coordinates) obtained in previous steps, the image region associated with the product is determined. These regions are typically the product itself or areas closely related to the product, including product labels, trademarks, packaging, and other product-related parts. By cropping or marking the portion of the target image containing the product, the precise location of the region to be processed is determined. This regional location information provides a clear spatial reference for subsequent feature extraction, analysis, and recognition.

[0104] Extract visual features from a local image of the area to be processed based on the position information of the area to be processed to determine visual feature information;

[0105] Specifically, once the regions to be processed are identified, visual features that aid in product identification are extracted from these regions. These local images are processed using deep learning models such as convolutional neural networks (CNNs) to extract the product's basic visual features, such as color, texture, shape, edges, and details. These features are typically extracted and abstracted gradually through multiple convolutional and pooling layers to generate feature vectors for product identification. These visual features provide the necessary information for subsequent product classification, matching, and further analysis.

[0106] Analyzing the relative positions of the commodities based on the commodity position information to determine spatial feature information;

[0107] Specifically, in this step, in addition to considering the visual characteristics of each product area, the relative positional relationships between products are also analyzed. For example, the products are related to each other in the image through distance, arrangement, or clustering. Using product location information (such as the coordinates of the bounding boxes), the spatial layout of the products is analyzed to identify their positional patterns in the image. This spatial feature can include information such as relative distance, arrangement order, and up-down, left-right relationships between products. This helps better understand the relationship between products in space and provides more context for subsequent product identification and sorting.

[0108] The visual feature information and the spatial feature information are weightedly fused to determine the product image feature information.

[0109] Specifically, the visual feature information extracted from the area to be processed is weightedly fused with the spatial feature information obtained by analysis. By designing a weighting mechanism, each feature is assigned a value according to its importance, and the visual information and spatial information are integrated into a unified product image feature representation. This fusion process can be achieved by simple weighted averaging or through feature fusion modules in deep learning (such as attention mechanism, weighted summation, feature splicing, etc.); through this weighted feature fusion, the obtained product image feature information is more comprehensive and accurate, providing strong support for product identification, classification and subsequent decision-making.

[0110] Based on pre-collected multi-source private data from smart vending scenarios, we fine-tune and optimize the open-source multimodal visual language model to obtain a large multimodal model for product recognition.

[0111] In one embodiment, fine-tuning and optimizing the open-source multimodal visual language model based on pre-collected multi-source private data in smart vending scenarios to obtain a large multimodal model for product recognition includes:

[0112] Based on the pre-collected multi-source raw data in the smart vending scenario, the multi-source raw data is cleaned and structured to obtain a labeled data set;

[0113] Specifically, multimodal raw data, including transaction records, surveillance videos, and static images of merchandise within the vending machines, is collected from multiple deployed vending machine terminals. Using unified data interfaces and format standards, key elements of each data type are extracted, such as the product area within the image frame, the product identification code within the transaction record, and the purchase timestamp. Subsequently, using a predefined labeling system, the image data is manually or semi-automatically annotated with information such as product category, quantity, and packaging appearance. Semantic normalization is also performed on textual data (such as product names and barcodes). Data cleaning removes invalid, ambiguous, or erroneous samples, completes structured processing, and generates an aligned image-text annotation dataset for model training, laying the foundation for subsequent model input.

[0114] Based on the image information and text labels in the annotated dataset, the image and text data are paired and constructed, and the format is converted and uniformly preprocessed to obtain a multimodal input sample set for training;

[0115] Specifically, for the annotated images and text, a semantic alignment algorithm is used to pair the image and text data one-to-one, ensuring that each pair of image and text samples has a clear semantic association (such as a product image and its product name or description text). To meet the input requirements of the multimodal pre-training model, the image data undergoes visual preprocessing operations such as size normalization, pixel value normalization, and data augmentation (such as random cropping and rotation). For text information, standardized processing such as word segmentation, encoding, truncation, and padding is performed based on the language encoder used by the model (such as the Transformer or BERT architecture). This ultimately generates a collection of image and text samples in a unified format that can be directly used as input for multimodal model training.

[0116] Based on the multimodal input sample set, we load weight parameters into the open-source visual language pre-training model and build a visual encoding and language encoding network structure that supports joint optimization to obtain the initial structure of the multimodal model for fine-tuning.

[0117] Specifically, open-source multimodal pre-trained models with image-text comprehension capabilities (such as CLIP and BLIP) were selected as the base model, their officially released pre-trained parameter weights were loaded, and some underlying modules were frozen to preserve their general representational capabilities. Furthermore, based on the characteristics of vending machines, a customized alignment strategy between visual encoders (such as ResNet or VisionTransformer) and language encoders (such as BERT or T5) was designed, introducing a cross-modal attention mechanism or contrastive learning structure to enhance the interactive expression capabilities between image and text semantics. The constructed model structure supports joint optimization of the parameters of the visual feature extraction module and the text semantic modeling module, exhibiting good fine-tunability and laying the structural foundation for subsequent multiple rounds of training on private datasets.

[0118] Fine-tuning the initial structure of the multimodal model according to the characteristics of the vending scenario and the recognition accuracy requirements, and optimizing the hyperparameter configuration through a cross-validation strategy to obtain multiple candidate multimodal models;

[0119] Specifically, to adapt to complex scenarios such as occlusion, varying angles, and low lighting in vending machine environments, a fine-tuning process involving multiple rounds of training was designed based on the initial structure of the multimodal model constructed in the third step. In each round of training, a strategy of freezing some backbone parameters and unfreezing specific high-level modules was introduced to improve the model's ability to learn domain-specific features. At the same time, a custom loss function was introduced to enhance the ability to distinguish similar products between categories. Through the K-fold cross-validation method, training experiments were repeated on different subsets of the private dataset, and hyperparameters such as learning rate, batch size, optimizer type, and regularization weight were systematically searched and adjusted. Ultimately, multiple candidate multimodal model versions with different performance emphases were output.

[0120] According to the preset accuracy, recall rate and response time, each candidate multimodal model is evaluated and compared to obtain the multimodal large model.

[0121] Specifically, to ensure the model's product recognition effectiveness and system response efficiency in actual deployment, a multi-dimensional performance evaluation framework was constructed, and the candidate models output from each round of fine-tuning were systematically compared on the test set. Accuracy and recall are used to measure the comprehensiveness and accuracy of the model in the product recognition task, and the F1 value is used as a comprehensive indicator to evaluate recognition stability; response time evaluates whether it can meet real-time recognition requirements after being embedded in the vending machine terminal. In addition, the model's recognition performance for similar products, occluded products, and combined products was further tested. Based on the above multiple indicators, the candidate models were ranked according to a preset weighted scoring system, and the model version with the best overall performance and satisfactory deployment performance was selected as the final multimodal large model for product recognition.

[0122] The commodity image feature information and the text information are input into the multimodal large model for information fusion, and the commodity target recognition result is determined based on the fused feature information.

[0123] Specifically, the previously extracted product image features are combined with the corresponding text information as multimodal input data. Image features contain visual elements (such as color, shape, and texture), while text provides a semantic description of the product, complementing each other. These image features and text information are then fed into a pre-trained multimodal large model. This model is typically based on the Transformer architecture or other deep learning structures, such as the CLIP model or VisualBERT. This model can simultaneously process image and text data and leverage the correlation between them for information fusion. In this multimodal large model, image and text features complement and align through shared representation learning. The model understands the relationship between the two, thereby forming a deep understanding of the product targets. Through this fusion of multimodal information, the model can more accurately identify product targets, such as product classification, brand recognition, and size matching, based on the fused information. It also enables efficient and precise product identification and location in complex commodity trading scenarios.

[0124] In one embodiment, see Figure 5 Inputting the product image feature information and the text information into the multimodal large model for information fusion, and determining the product target recognition result based on the fused feature information includes:

[0125] Inputting the product image feature information and the text information into the multimodal large model to obtain fused feature information that fuses image features and text semantics;

[0126] Specifically, a multimodal large model, such as the pre-trained CLIP or BLIP large model, is used to jointly model the product image feature information and the textual information. A multi-layer attention mechanism is used to extract the alignment information between the product image and the textual description. This fused feature vector is then output as fused information, which more comprehensively represents the multi-dimensional characteristic attributes of the product. This fused information can resolve semantic ambiguity caused by OCR misrecognition, fuzziness, and missing characters, while also addressing the inadequacy of image information in single-modal classification.

[0127] Inputting the fused feature information into a pre-trained product classification model to obtain an initial product category;

[0128] Specifically, the fusion feature information of the fused image features and text semantics is input into a pre-trained product classification model to obtain the initial product category. This product classification model is trained on a large amount of product image and text data, has a strong understanding of the fused semantics, and can output a high-confidence product category prediction result, including the category label and its corresponding confidence score. By leveraging the synergistic effect of multimodal semantics, the accuracy of the first-level product classification is improved, providing a clear category reference for subsequent judgment of whether similar products exist.

[0129] According to the initial product category, determining whether there are similar products in the current initial product category;

[0130] Specifically, a pre-set similar product mapping database is obtained, and according to the initial product category, whether there are product subcategories with highly similar visual / functional characteristics under the initial product category, such as Figure 6 As shown in the figure, similar products refer to products in the same series that are highly consistent in appearance elements such as core packaging design, bottle structure, material and main color, and are distinguished only by local identification differences (such as flavor text, pattern symbols or color block embellishments). Figure 6 For example, two drinks, both featuring white bottles and black caps with a minimalist, symmetrical design, belong to a product line under the same brand. Their similarity lies in the unified bottle shape, primary color scheme, and standardized brand logo layout; their differences lie in the flavor labels at the bottom of the bottles: a light gray peach logo with white peach text on the left, and a dark gray apple logo with green apple text on the right. This design strategy maintains brand visual consistency while enabling precise differentiation between product lines through subtle differences in graphics and text. This is a typical example of similar products with multiple flavors within the same product line, requiring refined identification and classification based on local features (such as specific graphics or text descriptions). The process of determining the presence of similar products can be performed through cluster analysis based on the knowledge graph, the product SKU attribute structure, or the Euclidean distance (or cosine similarity) between product image and text features. If the model confidence is low or there are multiple highly similar subcategories within the initial product category, similar products are considered within the initial product category, and the next step of the refined identification process is initiated. By automatically triggering a more accurate analysis process in uncertain or fuzzy classification scenarios, the judgment robustness and user experience in complex product recognition tasks are improved.

[0131] When similar products exist, obtaining a local area of ​​features to be extracted and a target feature to be extracted according to the initial product category;

[0132] Specifically, when similar products exist, salient regions are extracted from multiple sample images within the initial product category based on the initial product category. Using saliency detection models such as SAM, UNet-Saliency, GradCAM, and CLIP Attention, a saliency heatmap is generated to identify regions of highest visual interest, such as packaging edges, logos, unique patterns, and color blocks. By analyzing the similarities and differences between these salient regions across multiple sample images of similar products, a set of target features (such as bottle cap shape, edge lines, and background texture) is generated, and the coordinates of these localized regions are determined for subsequent detailed analysis. By focusing on truly valuable details that distinguish easily confused products, the classification model's ability to leverage image details is effectively enhanced, reducing misclassification rates and significantly improving recognition accuracy for key products, such as those with different flavors and sizes within the same brand.

[0133] In one embodiment, when similar products exist, obtaining the local area of ​​the feature to be extracted and the target feature to be extracted according to the initial product category includes:

[0134] According to the initial product category, sample images corresponding to multiple subcategories under the category are selected from a preset product image database;

[0135] Specifically, after initially determining the initial product category based on the fused feature information, the system searches for multiple subcategories corresponding to this initial product category in a pre-established product image database. Sample images corresponding to each subcategory are selected as the basis for subsequent analysis. These sample images cover common variations within this initial product category, such as product versions with different flavors, packaging styles, or specifications and capacities. By selecting these representative sample images, a rich and diverse visual sample resource is provided for subsequent differentiation of similar products, helping to further explore the differences in these products' image details.

[0136] Inputting each of the sample images into a pre-trained saliency detection model to obtain a saliency heat map, wherein the saliency heat map is used to characterize the area in the sample image where attention to visual features is most focused;

[0137] Specifically, the sample images of each subcategory obtained in the previous step are fed into a trained saliency detection model to automatically generate corresponding saliency heatmaps. This saliency detection model is trained on a large dataset of images with manually annotated salient regions. The model learns which image regions are most susceptible to feature extraction and establishes a mapping between the input image and the saliency distribution. During training, the model typically employs an encoder-decoder architecture, using a deep convolutional neural network to jointly model the image's global semantics and local texture. It then extracts and recovers the image's spatial features layer by layer to predict the saliency importance of each pixel. Training can incorporate ground truth fixations, boundary-preserving constraints, or contrastive loss functions to improve the model's ability to detect key regions in complex product images. This training process enables the model to automatically identify salient regions in unseen product images, providing a robust foundation for local region screening in subsequent product recognition. Saliency heatmaps highlight the most visually appealing regions in sample images. Commonly salient regions include brand logos, product names, color borders, and pattern elements. These areas are often the first parts that machine vision pays attention to when identifying products, and they are also an important reference for judging the differences in product details. Therefore, through saliency detection, accurate candidate ranges can be provided for subsequent local feature extraction and comparison.

[0138] Performing threshold segmentation on the saliency heat map to obtain multiple candidate regions;

[0139] Specifically, after obtaining the saliency heatmap, a pre-set threshold strategy is used to segment the saliency heatmap. Regions with saliency values ​​exceeding the preset segmentation threshold are extracted from the heatmap to generate several candidate regions. In general, the saliency value is a numerical value in the saliency heatmap that indicates the degree of attention drawn to each pixel or region in the image. The value typically ranges from zero to one, with higher values ​​indicating more pronounced features at that location. In product images, regions with high saliency values ​​are often concentrated in locations such as brand logos, textual content, and pattern details. Therefore, the saliency value can serve as an important indicator of visual focus in an image, helping to identify local areas that require subsequent analysis. These candidate regions represent highly salient information segments in the sample image that may contain key visual features. This approach effectively narrows the analysis scope, focusing on the most distinctive and recognizable regions in the product image, providing targeted image segments for subsequent local feature analysis and scoring, thereby improving recognition efficiency and accuracy.

[0140] Performing a comprehensive scoring on each of the candidate regions, and screening the local region from each of the candidate regions according to the scoring results;

[0141] Specifically, after candidate regions are extracted, each candidate region is evaluated using a comprehensive, multi-dimensional score. Scoring metrics include the region's average saliency value in the saliency heatmap and its response strength relative to product category relevance. Specifically, category activation information is further obtained based on the initial product category, and the candidate region's performance in the response heatmap is evaluated. The saliency score is then weighted and fused with the category-related score to arrive at a final score for each candidate region. Ultimately, based on the scoring results, candidate regions with high discriminative power are selected as local regions for subsequent target feature extraction and identification of similar products.

[0142] In one embodiment, the comprehensive scoring of the candidate regions and obtaining the local region according to the scoring results include:

[0143] Obtain the saliency value in the saliency heat map corresponding to each candidate region;

[0144] Specifically, candidate regions are multiple regions of interest preliminarily selected on the heat map through methods such as threshold segmentation. The saliency values ​​of all pixels contained in each candidate region in the saliency heat map are extracted. These saliency values ​​represent the degree of visual attention of the region in the entire image and are the core basis for measuring the saliency of the region.

[0145] Calculating the average significance value of each candidate region as a significance score value according to each of the significance values;

[0146] Specifically, the saliency values ​​within each candidate region are averaged to obtain a unified value as the candidate region's saliency score. This score reflects the overall visual appeal of the region. A higher score indicates that the region is more likely to be the first focus in most sample images, thus having higher analysis and recognition value. This score will serve as one of the important indicators for subsequent region screening.

[0147] According to the initial product category, each of the sample images is input into a pre-trained image classification model to obtain a category activation map;

[0148] Specifically, to further assess the association between candidate regions and product categories, each sample image is fed into a trained image classification model. During the recognition process, the model generates a class activation map. The image classification model is trained on a large dataset of labeled product images and employs a deep convolutional neural network architecture, such as a residual network or an efficient network architecture. By continuously adjusting weight parameters, the model accurately maps input images to predefined product category labels. During training, the model not only learns the correspondence between the overall image and the category, but also implicitly learns which regions within the image best represent the category. The class activation map is generated based on this mechanism. When a sample image is fed into the model, feature channels associated with the target category are extracted from the last convolutional feature map layer of the image classification model. These channels are weighted and summed before being projected onto the image space, resulting in a class activation map. This class activation map identifies image regions that strongly respond to the initial product category, typically covering key identification elements such as brand name, product structure, and key patterns. This further validates and reinforces the regional information provided by the saliency heatmap.

[0149] Obtaining a response heat map of each sample image under the current initial product category according to the category activation map;

[0150] Specifically, the category activation map is refined into a response heatmap associated with the subcategories corresponding to the initial product category. This response heatmap focuses on the regions that most stimulate the classification model's response within the current category. By extracting the corresponding pixel values ​​of the candidate regions in this response heatmap, the correlation between the candidate regions and the current product category can be measured, laying the foundation for the subsequent calculation of the category relevance score. This step helps identify which candidate regions are not only significant but also highly relevant to the product recognition task itself, thereby improving the final recognition accuracy.

[0151] Counting the pixels at the corresponding position of each candidate region in the response heat map, and calculating the average activation intensity of each candidate region as a category-related score value;

[0152] Specifically, the category activation map is aligned with the position of each candidate region, and the corresponding pixel values ​​of the candidate region in the response heat map are extracted. These pixel values ​​reflect the degree of response of the region to the initial product category in the image classification model. By averaging all the response values ​​within the candidate region, the category-related score value of the candidate region is obtained, which represents the importance of the region to the target category. The advantage of this score is that it does not rely on manual judgment, but is derived from the internal feature response of the deep model. It effectively quantifies the true correlation between each region in the image and the category, which helps to accurately select local regions with category discrimination in the subsequent process.

[0153] Performing weighted fusion on the significance score value of each candidate region and the category-related score value to obtain the score result of each candidate region;

[0154] Specifically, to comprehensively assess the salience and category relevance of each candidate region, the salience score and the category relevance score are weighted and fused. The weighting strategy can be set based on the importance of the task; for example, a higher weight can be given to the category relevance score to highlight its role in product differentiation. The final fused score represents the comprehensive recognition value of each candidate region, taking into account both the attention bias (salience) and the model's classification focus (category relevance). This fusion mechanism achieves dual perceptual and semantic guidance, effectively improving the accuracy and practicality of local feature extraction.

[0155] The scoring results are compared with a preset scoring threshold, and according to the comparison results, at least one region is selected from each candidate region as the local region.

[0156] Specifically, a pre-set scoring threshold is obtained to filter out candidate regions with low scores and no recognition significance. The fusion scoring result is then compared against the scoring threshold one by one, retaining only regions with scores above the threshold as valid local regions. If multiple candidate regions meet the criteria, multiple local regions can be selected for subsequent processing. This step, using the threshold screening mechanism, avoids interference from noisy regions and improves the targeted nature of feature extraction. Selecting multiple high-scoring regions preserves richer product details, providing a more complete feature base for subsequent refined comparison and difference analysis.

[0157] Candidate feature extraction and feature evaluation are performed on the local area, and the target feature is obtained by screening from the extracted candidate features based on the feature evaluation result.

[0158] Specifically, a trained feature extraction network (such as one based on a convolutional or visual Transformer architecture) is used to perform deep image feature extraction on each local area. The extracted feature vector contains multi-dimensional information such as color, shape, texture, and pattern, forming multiple candidate features. These candidate features are then evaluated, for example, by performing similarity comparisons and difference analysis with standard features of products in the same category, or by scoring them using a pre-trained feature quality scoring model, to select the most representative target features that can best distinguish the product. The advantage of this process is that it can automatically extract the most useful key features for product identification from complex images, which not only improves classification accuracy but also supports efficient application in subsequent retrieval, comparison, and recommendation scenarios.

[0159] In one embodiment, the extracting and evaluating candidate features of the local area, and screening the target feature from the extracted candidate features according to the feature evaluation result, includes:

[0160] According to the initial product category, obtaining a feature extraction strategy corresponding to each candidate feature corresponding to the category;

[0161] Specifically, after identifying the initial product category of the current product image (e.g., carbonated beverages, juice drinks, functional drinks, lactic acid bacteria drinks), etc.), the system automatically retrieves a feature extraction scheme specifically configured for that category from a pre-set feature extraction strategy library. This strategy summarizes the visual structural differences among beverage products. For example, in the carbonated beverage category, the strategy prioritizes the extraction of color block boundaries, font style features (such as prominent brand name), bubble texture, or bottle cap shape. In contrast, in the juice beverage category, the feature extraction strategy emphasizes the color saturation of the fruit image, the juice content information on the bottle label, and the degree of realism of the image. Each feature extraction strategy defines parameters such as the spatial location, scale, priority channel, and receptive field (achievable by adjusting the convolution kernel size or network layer level) of the feature extraction. These parameters guide the subsequent targeted multi-path feature extraction within the local area. The advantage of this strategy design is that it automatically adapts to the most discernible image features across different beverage categories, thereby improving the system's ability to distinguish similar products within a broad category and reducing the risk of misclassification due to similar styles.

[0162] Performing multi-path feature extraction on the local area according to each of the feature extraction strategies to obtain a plurality of candidate feature information;

[0163] Specifically, based on the selected feature extraction strategy, multi-path feature extraction is performed on the local areas screened in the previous step. The so-called multi-path refers to the parallel analysis of the image content of the local area from multiple angles and scales through different neural network branches or parameter paths (such as multi-scale convolution, different channel attention paths, shape feature paths, etc.). Each path outputs a set of feature representations, ultimately obtaining multiple candidate feature information. The candidate feature information includes multi-dimensional data such as local texture, edge contours, color distribution patterns, pattern arrangement, etc. This multi-path approach can maximize the capture of detailed changes in the image that may be used for classification or comparison, improving the system's ability to represent complex product images and the robustness of subsequent analysis.

[0164] Performing a distribution consistency analysis on each candidate feature information in each sample image, and obtaining the frequency and position deviation of each candidate feature information in different sample images as a distribution consistency indicator;

[0165] Specifically, in order to further determine the stability and representativeness of candidate features, a distribution consistency analysis is performed on the same candidate feature information extracted from multiple sample images. Distribution consistency analysis includes: counting the frequency of the candidate feature appearing in all sample images, that is, whether it can stably appear in most samples; calculating the position deviation of the candidate in the image, such as the degree of difference in the coordinates of its center point in each image. Features with high frequency and small position deviation usually indicate strong commonality and stability in the product category, and are therefore given a higher distribution consistency index score. This indicator not only helps to screen out noise or invalid features that appear accidentally, but also improves the accuracy and interpretability of subsequent target feature screening.

[0166] Input each candidate feature information into a pre-trained product recognition model to obtain a recognition result, and obtain a classification confidence corresponding to the recognition result as a classification response strength of each candidate feature information;

[0167] Specifically, each candidate feature information is input one by one into a product recognition model that has been pre-trained with a large-scale product dataset. This model is usually based on a deep convolutional neural network architecture and can automatically extract multi-level semantic representations from the input local features and perform fine category discrimination. While outputting the recognition results, the model will also give the corresponding classification confidence as the classification response strength of each candidate feature information. This confidence reflects the model's degree of confidence in the judgment of the product category corresponding to the candidate feature. The classification confidence not only reflects the recognition ability of the feature, but also quantifies the contribution of the feature to the overall classification decision, providing a scientific basis for subsequent feature screening.

[0168] Evaluate each candidate feature information according to the distribution consistency index and the classification response strength to obtain a feature evaluation result;

[0169] Specifically, a comprehensive evaluation is conducted by combining the distribution consistency index and classification response strength of candidate features. The distribution consistency index measures the frequency of occurrence and spatial position stability of a feature across samples within the same category, ensuring that the feature is not accidental information caused by chance or noise, but truly represents the common characteristics of the category. The classification response strength, on the other hand, measures the influence and importance of the feature on the classification result. By designing a weighted fusion or multidimensional scoring mechanism, these two indicators are comprehensively considered to obtain a feature evaluation result, avoiding misjudgments caused by a single dimension. This results in a more objective and comprehensive feature quality assessment and ensures that the selected features are both stable and effective.

[0170] According to the feature evaluation result, the target feature is screened out from each candidate feature information.

[0171] Specifically, based on the above feature evaluation results, target features are screened from all candidate feature information. During the screening process, features with high distribution consistency and high classification response strength are prioritized. These target features have demonstrated good stability and discriminability across multiple samples. The screening of target features not only improves the accuracy of product category differentiation but also effectively supports subsequent similar product identification and fine-grained analysis. This screening step significantly improves the ability to capture detailed differences in beverage products, enhancing the generalization performance and practical value of the overall recognition model.

[0172] Extracting features of the target image based on the local region and the target features to obtain local region feature information;

[0173] Specifically, based on the previously screened local regions and target features, a refined feature extraction operation is performed on the target image. By extracting features from the target image, multidimensional information containing the target features is extracted from the local region of the target image. This multidimensional information includes color distribution, texture details, shape structure, and the spatial relationships of feature points. Through deep learning techniques such as multi-path convolutional neural networks or feature pyramid networks, it is possible to fully capture the detailed features of the local region that are of discernible value, ensuring that the extracted local region feature information is both rich and representative, thereby providing accurate and high-quality input data for subsequent product classification.

[0174] The initial commodity category is classified according to the local area feature information to obtain a target commodity category as the commodity target recognition result.

[0175] Specifically, the extracted local region feature information is input into a fine-grained classification model pre-trained for the initial product category for classification. This classification model is specifically designed to process local features and can fully exploit the subtle feature differences between different beverage categories, thereby accurately distinguishing subcategories within the initial product category. The model enhances its perception of key local features through multi-layer nonlinear transformations and attention mechanisms, ultimately outputting the target product category as the product target recognition result. This process not only improves classification accuracy and robustness, but also effectively reduces the risk of misjudgment caused by complex or diverse backgrounds in product images, achieving efficient and accurate product target recognition.

[0176] Example 2

[0177] See Figure 7 Embodiment 2 of the present invention further provides a multi-target commodity recognition device based on multimodal data processing, the device comprising:

[0178] A real-time image acquisition module is used to acquire real-time video data in a commodity trading scenario and decompose the real-time video data into multiple frames of real-time images;

[0179] A preprocessing and label information extraction module, configured to preprocess the real-time image and extract label information, and determine text information corresponding to the preprocessed target image and the product label;

[0180] An instance segmentation module, configured to perform instance segmentation on the target image to determine product location information;

[0181] A feature extraction module is used to extract features of the target image based on the product location information to determine product image feature information;

[0182] The multimodal large model training module is used to fine-tune and optimize the open source multimodal visual language model based on pre-collected multi-source private data in smart vending scenarios, thereby obtaining a multimodal large model for product recognition.

[0183] The commodity recognition module is used to input the commodity image feature information and the text information into the multimodal large model for information fusion, and determine the commodity target recognition result based on the fused feature information.

[0184] Specifically, a multi-target commodity recognition device based on multimodal data processing provided by Example 2 of the present invention is adopted, and the device includes: a real-time image acquisition module, which is used to acquire real-time video data in a commodity trading scenario and decompose the real-time video data into multiple frames of real-time images; a preprocessing and label information extraction module, which is used to preprocess and extract label information on the real-time image, and determine the text information corresponding to the preprocessed target image and the commodity label; an instance segmentation module, which is used to perform instance segmentation on the target image and determine the commodity location information; a feature extraction module, which is used to extract features of the target image based on the commodity location information and determine the commodity image feature information; a multimodal large model training module, which is used to fine-tune and optimize the open source multimodal visual language model based on pre-collected multi-source privatized data in the smart vending scenario to obtain a multimodal large model for commodity recognition; a commodity recognition module, which is used to input the commodity image feature information and the text information into the multimodal large model for information fusion, and determine the commodity target recognition result based on the fused feature information. This device extracts multiple frames of images from real-time video data, and through preprocessing and label information extraction, it preliminarily identifies the products in each frame of the image, extracts the product labels and corresponding text information, then processes the target image through instance segmentation technology, accurately locates the bounding box of each product and solves the occlusion problem between products, and uses the product location information to further extract image features, including visual features, spatial features and semantic features. Through multimodal large models, information fusion is performed to combine the product image features with the extracted text information to improve the recognition ability of complex product targets. Finally, based on the fusion of image and text information, the model can accurately distinguish and identify multiple product targets, and maintain efficient recognition performance even in complex environments where multiple products exist at the same time and there is partial overlap or occlusion. It not only improves the recognition accuracy, but also can operate stably in dynamic scenes to meet the needs of multi-target recognition.

[0185] Example 3

[0186] In addition, Example 3 of the present invention also provides a multi-target commodity identification system based on multimodal data processing, including: an image acquisition device, at least one processor, at least one memory, and computer program instructions stored in the memory. When the computer program instructions are executed by the processor, the method described in Example 1 is implemented.

[0187] Figure 8 A schematic diagram of the hardware structure of an electronic device provided in Example 3 of the present invention is shown.

[0188] An electronic device may include a processor and a memory storing computer program instructions.

[0189] Specifically, the processor may include a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits for implementing the embodiments of the present invention.

[0190] The memory may include a large-capacity storage for data or instructions. By way of example, and not limitation, the memory may include a hard disk drive (HDD), a floppy disk drive, flash memory, an optical disk, a magneto-optical disk, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory may include removable or non-removable (or fixed) media. Where appropriate, the memory may be internal or external to the data processing device. In certain embodiments, the memory is non-volatile solid-state memory. In certain embodiments, the memory includes read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or flash memory, or a combination of two or more of these.

[0191] The processor reads and executes computer program instructions stored in the memory to implement any one of the multi-target commodity recognition methods based on multimodal data processing in the above embodiments.

[0192] In one example, the electronic device may further include a communication interface and a bus. Figure 8 As shown, the processor, memory, and communication interface are connected via a bus and communicate with each other.

[0193] The communication interface is mainly used to implement communication between the modules, devices, units and / or equipment in the embodiments of the present invention.

[0194] The bus includes hardware, software or both, and couples the components of the device to each other. By way of example and not limitation, the bus may include an accelerated graphics port (AGP) or other graphics bus, an enhanced industry standard architecture (EISA) bus, a front side bus (FSB), a hypertransport (HT) interconnect, an industry standard architecture (ISA) bus, an infinite bandwidth interconnect, a low pin count (LPC) bus, a memory bus, a microchannel architecture (MCA) bus, a peripheral component interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a serial advanced technology attachment (SATA) bus, a video electronics standard association local (VLB) bus or other suitable buses or a combination of two or more of these. Where appropriate, the bus may include one or more buses. Although the embodiments of the present invention describe and illustrate specific buses, the present invention contemplates any suitable bus or interconnect.

[0195] In summary, the embodiments of the present invention provide a method, device, and system for multi-target commodity recognition based on multimodal data processing.

[0196] It should be understood that the present invention is not limited to the specific configurations and processes described above and illustrated in the figures. For the sake of brevity, a detailed description of known methods is omitted. In the above embodiments, several specific steps are described and illustrated as examples. However, the method of the present invention is not limited to the specific steps described and illustrated. Those skilled in the art may make various changes, modifications, and additions, or change the order of the steps after understanding the spirit of the present invention.

[0197] The functional blocks shown in the block diagrams described above can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they may be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, and the like. When implemented in software, the elements of the present invention are programs or code segments used to perform the desired tasks. Programs or code segments may be stored in a machine-readable medium or transmitted via a data signal carried in a carrier wave over a transmission medium or communication link. "Machine-readable medium" may include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROMs, flash memory, erasable ROMs (EROMs), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, and the like. Code segments may be downloaded via a computer network such as the Internet or an intranet.

[0198] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant location, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0199] It should also be noted that the exemplary embodiments described herein describe methods or systems based on a series of steps or devices. However, the present invention is not limited to the order of the steps described above. In other words, the steps may be performed in the order described in the embodiments, or in a different order, or several steps may be performed simultaneously.

[0200] The above description is only a specific embodiment of the present invention. Those skilled in the art will clearly understand that for the convenience and brevity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present invention is not limited to this. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention.

Claims

1. A multi-target commodity recognition method based on multimodal data processing, characterized in that: The method comprises: Acquire real-time video data in a commodity trading scenario, and decompose the real-time video data into multiple frames of real-time images; Preprocessing and label information extraction are performed on the real-time image to determine text information corresponding to the preprocessed target image and the product label; Performing instance segmentation on the target image to determine product location information; Extracting features of the target image based on the product location information to determine product image feature information; Based on pre-collected multi-source private data from smart vending scenarios, we fine-tune and optimize the open-source multimodal visual language model to obtain a large multimodal model for product recognition. Inputting the product image feature information and the text information into the multimodal large model for information fusion, and determining a product target recognition result based on the fused feature information; Inputting the product image feature information and the text information into the multimodal large model for information fusion, and determining the product target recognition result based on the fused feature information includes: Inputting the product image feature information and the text information into the multimodal large model to obtain fused feature information that fuses image features and text semantics; Inputting the fused feature information into a pre-trained product classification model to obtain an initial product category; According to the initial product category, determining whether there are similar products in the current initial product category; When similar products exist, obtaining a local area of ​​features to be extracted and a target feature to be extracted according to the initial product category; Extracting features of the target image based on the local region and the target features to obtain local region feature information; Classifying the initial commodity category according to the local area feature information to obtain a target commodity category as the commodity target recognition result; When similar products exist, obtaining the local area of ​​the feature to be extracted and the target feature to be extracted according to the initial product category includes: According to the initial product category, sample images corresponding to multiple subcategories under the category are selected from a preset product image database; Inputting each of the sample images into a pre-trained saliency detection model to obtain a saliency heat map, wherein the saliency heat map is used to characterize the area in the sample image where attention to visual features is most focused; Performing threshold segmentation on the saliency heat map to obtain multiple candidate regions; Performing a comprehensive scoring on each of the candidate regions, and screening the local region from each of the candidate regions according to the scoring results; Candidate feature extraction and feature evaluation are performed on the local area, and the target feature is obtained by screening from the extracted candidate features based on the feature evaluation result.

2. The multi-target commodity identification method based on multimodal data processing according to claim 1, characterized in that: The preprocessing of the real-time image and the extraction of label information to determine text information corresponding to the preprocessed target image and the product label includes: Performing size adjustment and noise reduction processing on the real-time image to determine the target image; Performing target detection on the target image to determine product area location information; According to the commodity area location information, commodity labels in the commodity area are processed using optical character recognition technology to determine the text information.

3. The multi-target commodity identification method based on multimodal data processing according to claim 2, characterized in that: The performing instance segmentation on the target image to determine the commodity location information includes: Extracting feature information of the product area through a convolutional neural network based on the product area location information, and determining a candidate area based on the extracted feature information; Processing the candidate regions through an instance segmentation network to determine a binary image corresponding to each commodity target; The binary image is processed using post-processing technology to determine the commodity location information.

4. The multi-target commodity identification method based on multimodal data processing according to claim 1, characterized in that: The open source multimodal visual language model is fine-tuned and optimized based on pre-collected multi-source private data in the smart vending scenario to obtain a large multimodal model for product recognition, including: Based on the pre-collected multi-source raw data in the smart vending scenario, the multi-source raw data is cleaned and structured to obtain a labeled data set; Based on the image information and text labels in the annotated dataset, the image and text data are paired and constructed, and the format is converted and uniformly preprocessed to obtain a multimodal input sample set for training; Based on the multimodal input sample set, we load weight parameters into the open-source visual language pre-training model and build a visual encoding and language encoding network structure that supports joint optimization to obtain the initial structure of the multimodal model for fine-tuning. Fine-tuning the initial structure of the multimodal model according to the characteristics of the vending scenario and the recognition accuracy requirements, and optimizing the hyperparameter configuration through a cross-validation strategy to obtain multiple candidate multimodal models; According to the preset accuracy, recall rate and response time, each candidate multimodal model is evaluated and compared to obtain the multimodal large model.

5. The multi-target commodity identification method based on multimodal data processing according to claim 1, characterized in that: The candidate regions are comprehensively scored, and the local regions obtained according to the scoring results include: Obtain the saliency value in the saliency heat map corresponding to each candidate region; Calculating the average significance value of each candidate region as a significance score value according to each of the significance values; According to the initial product category, each of the sample images is input into a pre-trained image classification model to obtain a category activation map; Obtaining a response heat map of each sample image under the current initial product category according to the category activation map; Counting the pixels at the corresponding position of each candidate region in the response heat map, and calculating the average activation intensity of each candidate region as a category-related score value; Performing weighted fusion on the significance score value of each candidate region and the category-related score value to obtain the score result of each candidate region; The scoring results are compared with a preset scoring threshold, and according to the comparison results, at least one region is selected from each candidate region as the local region.

6. The multi-target commodity identification method based on multimodal data processing according to claim 1, characterized in that: The extracting and evaluating candidate features of the local area, and screening the target feature from the extracted candidate features according to the feature evaluation result, includes: According to the initial product category, obtaining a feature extraction strategy corresponding to each candidate feature corresponding to the category; Performing multi-path feature extraction on the local area according to each of the feature extraction strategies to obtain a plurality of candidate feature information; Performing a distribution consistency analysis on each candidate feature information in each sample image, and obtaining the frequency and position deviation of each candidate feature information in different sample images as a distribution consistency indicator; Input each candidate feature information into a pre-trained product recognition model to obtain a recognition result, and obtain a classification confidence corresponding to the recognition result as a classification response strength of each candidate feature information; Evaluate each candidate feature information according to the distribution consistency index and the classification response strength to obtain a feature evaluation result; According to the feature evaluation result, the target feature is screened out from each candidate feature information.

7. A multi-target commodity recognition device based on multimodal data processing, characterized in that: The device comprises: A real-time image acquisition module is used to acquire real-time video data in a commodity trading scenario and decompose the real-time video data into multiple frames of real-time images; A preprocessing and label information extraction module, configured to preprocess the real-time image and extract label information, and determine text information corresponding to the preprocessed target image and the product label; An instance segmentation module, configured to perform instance segmentation on the target image to determine product location information; A feature extraction module is used to extract features of the target image based on the product location information to determine product image feature information; The multimodal large model training module is used to fine-tune and optimize the open source multimodal visual language model based on pre-collected multi-source private data in smart vending scenarios, thereby obtaining a multimodal large model for product recognition. A commodity recognition module is used to input the commodity image feature information and the text information into the multimodal large model for information fusion, and determine the commodity target recognition result based on the fused feature information; Inputting the product image feature information and the text information into the multimodal large model for information fusion, and determining the product target recognition result based on the fused feature information includes: Inputting the product image feature information and the text information into the multimodal large model to obtain fused feature information that fuses image features and text semantics; Inputting the fused feature information into a pre-trained product classification model to obtain an initial product category; According to the initial product category, determining whether there are similar products in the current initial product category; When similar products exist, obtaining a local area of ​​features to be extracted and a target feature to be extracted according to the initial product category; Extracting features of the target image based on the local region and the target features to obtain local region feature information; Classifying the initial commodity category according to the local area feature information to obtain a target commodity category as the commodity target recognition result; When similar products exist, obtaining the local area of ​​the feature to be extracted and the target feature to be extracted according to the initial product category includes: According to the initial product category, sample images corresponding to multiple subcategories under the category are selected from a preset product image database; Inputting each of the sample images into a pre-trained saliency detection model to obtain a saliency heat map, wherein the saliency heat map is used to characterize the area in the sample image where attention to visual features is most focused; Performing threshold segmentation on the saliency heat map to obtain multiple candidate regions; Performing a comprehensive scoring on each of the candidate regions, and screening the local region from each of the candidate regions according to the scoring results; Candidate feature extraction and feature evaluation are performed on the local area, and the target feature is obtained by screening from the extracted candidate features based on the feature evaluation result.

8. A multi-target commodity recognition system based on multimodal data processing, characterized in that: include: An image acquisition device, at least one processor, at least one memory, and computer program instructions stored in the memory, which implement the method according to any one of claims 1 to 6 when the computer program instructions are executed by the processor.

Images