An article identification method, device, electronic equipment and readable storage medium

Abstract

The application discloses an article identification method and device, electronic equipment and readable storage medium. The method comprises the following steps: collecting a first image of an article to be identified, and determining the interference degree and type of the first image caused by condensed water; determining a target imaging mode from a first imaging mode, a second imaging mode and a third imaging mode according to the interference degree and the interference type; the first imaging mode is a mode for collecting an image under visible light irradiation, the second imaging mode is a mode for collecting an image under invisible light irradiation, and the third imaging mode is a mode for collecting an image under the condition that a polarizer is arranged at at least two different angles; collecting a second image of the article to be identified in the target imaging mode, and determining an article category of the article to be identified based on a target feature vector corresponding to the second image; in the case of various complex condensed water interference, high-robustness features are extracted, and the accuracy and stability of article identification based on the features are improved.

Description

Technical Field

[0001] This application belongs to the field of computer technology, and specifically relates to an item identification method, device, electronic device, and readable storage medium. Background Technology

[0002] With the development of the Internet of Things (IoT) and artificial intelligence (AI) systems, smart retail devices such as unmanned vending machines and automated vending machines are typically equipped with image recognition systems. These systems collect images of products inside the vending machine and use computer vision technology to identify product categories. In actual use, condensation can easily form on the inside of the vending machine door glass or on the surface of the products, severely interfering with image acquisition quality and reducing recognition accuracy.

[0003] Currently, post-processing algorithms are commonly used to mitigate condensation interference. This involves applying software enhancement, dehazing, or deep learning to the damaged image after condensation has formed, restoring obscured or blurred product features. However, condensation absorbs and scatters light, causing irreversible loss of crucial information such as edges, textures, and colors of the original object. Therefore, under severe condensation interference, the image recognition rate after post-processing algorithm restoration still drops significantly. Alternatively, heated glass can be installed inside the cabinet to prevent condensation, or near-infrared imaging can be used to attempt to penetrate the water vapor. However, these methods are only effective for specific types of condensation, lack adaptability, and struggle to maintain stability in various complex environments.

[0004] Therefore, the loss of original image information caused by condensation cannot be fundamentally recovered by post-processing algorithms. The recognition accuracy is low under severe condensation interference, and it cannot adapt to complex condensation environments, resulting in low stability. Summary of the Invention

[0005] The purpose of this application is to provide an item identification method, device, electronic device, and readable storage medium that can still extract highly robust features even under various complex condensation interferences, thereby improving the accuracy and stability of feature-based item identification.

[0006] In a first aspect, embodiments of this application provide an article identification method, the method comprising: Acquire a first image of the object to be identified, and determine the degree and type of interference caused by condensation in the first image; Based on the degree and type of interference, a target imaging mode is determined from a first imaging mode, a second imaging mode, and a third imaging mode; the first imaging mode is a mode for acquiring images under visible light illumination, the second imaging mode is a mode for acquiring images under invisible light illumination, and the third imaging mode is a mode for acquiring images with the polarizer set at at least two different angles. A second image of the object to be identified is acquired in the target imaging mode, and the object category of the object to be identified is determined based on the target feature vector corresponding to the second image.

[0007] Secondly, embodiments of this application provide an item identification device, the device comprising: The image evaluation module is used to acquire a first image of the object to be identified and to determine the degree and type of interference caused by condensation in the first image. An imaging mode determination module is used to determine a target imaging mode from a first imaging mode, a second imaging mode, and a third imaging mode based on the interference level and the interference type; the first imaging mode is a mode for acquiring images under visible light illumination, the second imaging mode is a mode for acquiring images under invisible light illumination, and the third imaging mode is a mode for acquiring images with the polarizer set at at least two different angles. The category determination module is used to acquire a second image of the object to be identified in the target imaging mode, and determine the object category of the object to be identified based on the target feature vector corresponding to the second image.

[0008] Thirdly, embodiments of this application provide an electronic device including a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the method described in the first aspect.

[0009] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0010] Fifthly, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the method as described in the first aspect.

[0011] In this embodiment, a first image is first acquired, and the degree and type of interference from condensation water are determined. Then, based on the degree and type of interference, a target imaging mode is determined from a first imaging mode, a second imaging mode, and a third imaging mode. A second image is then acquired under the target imaging mode, and features are extracted for identification. By bringing the decision-making process forward, when the interference level is high, the contaminated visible light channel in a single imaging mode is actively discarded, and the acquisition is switched to a physical dimension less affected by condensation water, such as a second imaging mode utilizing invisible light or a third imaging mode using a polarizer with multiple angles. Specifically, the invisible light imaging mode utilizes the strong penetrability of near-infrared light through the water film to obtain geometric features from the spectral dimension; the polarizer multi-angle imaging mode utilizes the ability of polarization difference to suppress specular reflection to obtain enhanced texture features from the polarization dimension. These two modes avoid interference from condensation water at the source of acquisition, and even when severe condensation water causes significant loss of visible light image information, highly robust feature vectors can still be extracted. The system determines the target imaging mode based on the degree and type of interference. In other words, it can identify the current level and distribution of condensate interference in real time and adaptively select the most effective physical imaging mode accordingly. This automatic matching of the optimal image acquisition scheme for different condensate morphologies ensures stable and high-precision recognition performance even in complex and changing environments, significantly improving the overall adaptability of the system. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a flowchart illustrating the steps of an embodiment of an item identification method provided by the present invention; Figure 2 This is a schematic diagram of the architecture of an item recognition system provided by the present invention; Figure 3 This is a flowchart illustrating the steps of another embodiment of the item recognition method provided by the present invention; Figure 4 This is a structural block diagram of an item recognition device provided by the present invention; Figure 5 This is a structural block diagram of an electronic device provided by the present invention. Detailed Implementation

[0014] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0015] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0016] The terms "first," "second," etc., used in the specification and claims of this invention are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, the first object can be one or more. Furthermore, the term "and / or" in the specification and claims is used to describe the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. In embodiments of this invention, the term "multiple" refers to two or more, and other quantifiers are similar.

[0017] Method Implementation Examples Reference Figure 1 The diagram illustrates a flowchart of an embodiment of an item recognition method according to the present invention, the method comprising: Step 101: Acquire a first image of the object to be identified, and determine the degree and type of interference caused by condensation in the first image; Step 102: Determine the target imaging mode from the first imaging mode, the second imaging mode, and the third imaging mode according to the interference level and the interference type; the first imaging mode is the mode of acquiring images under visible light illumination, the second imaging mode is the mode of acquiring images under invisible light illumination, and the third imaging mode is the mode of acquiring images with the polarizer set at at least two different angles. Step 103: Acquire a second image of the object to be identified in the target imaging mode, and determine the object category of the object to be identified based on the target feature vector corresponding to the second image.

[0018] Regarding steps 101 to 103, the item recognition method proposed in this application is mainly applied to intelligent retail devices such as unmanned vending machines and automated vending machines. Condensation easily forms on the inside of the cabinet door glass or on the surface of the product packaging of such intelligent retail devices, severely interfering with image recognition.

[0019] The items to be identified refer to goods placed inside the vending machine, such as beverages, snacks, and daily necessities. The first image refers to the color image captured by the system in its initial state using visible light illumination, used to assess whether there is condensation interference in the current environment. Condensation refers to moisture that condenses on the glass surface inside the vending machine or on the surface of the goods due to temperature differences, and can appear as a continuous water film or discrete water droplets.

[0020] Interference level refers to a quantitative indicator measuring the severity of the impact of condensation on image quality, such as light, medium, and heavy levels. Interference type describes the spatial distribution of condensation within the imaging area, such as continuous uniform distribution or discrete non-uniform distribution. The first imaging mode refers to the mode that uses visible light as the illumination source to acquire a conventional color image of the object to be identified. The second imaging mode refers to the mode that uses invisible light as the illumination source to acquire an image of the object to be identified, such as near-infrared light. It should be noted that invisible light has a strong penetrating ability through water films. The third imaging mode refers to the mode that controls the polarizer in front of the image sensor to switch to at least two different angles, acquires images separately, and performs subsequent processing. The third imaging mode can suppress specular reflection caused by discrete water droplets.

[0021] It should be noted that visible light refers to the wavelength range visible to the human eye, such as 400nm-700nm, and is used for conventional color imaging. Invisible light refers to the wavelength range invisible to the human eye, such as 940nm near-infrared light. This wavelength of invisible light can penetrate water films without interfering with the viewer's vision. A polarizer is a liquid crystal device placed in front of the image sensor that can rapidly change its polarization direction under the drive of an electrical signal, used to select light with a specific vibration direction to pass through.

[0022] The second image refers to an image acquired under a defined target imaging pattern. Compared to the distorted first image, the second image contains more reliable product information. The target feature vector refers to the features extracted from the second image that characterize product attributes, such as geometric shape features or surface texture features. The item category, or final identification result, refers to the product's unique identifier, such as a Stock Keeping Unit (SKU).

[0023] It should be noted that the item recognition method provided in this application embodiment can be applied to an item recognition system, as shown below. Figure 2This diagram illustrates the framework of an object recognition system. The system comprises a hardware system, a core intelligent algorithm, and an application layer. The hardware system includes a multispectral active illumination module, an adjustable polarization image sensing module, an environmental perception module, and a central processing and control module. The multispectral active illumination module contains a standard visible light emitting diode (LED) array and a near-infrared structured light projector. The standard visible light LED array is used for conventional imaging, while the near-infrared structured light projector is used for invisible light imaging and can project coded patterns. The adjustable polarization image sensing module includes a global shutter CMOS sensor and an electronically controlled liquid crystal polarizer integrated in front of the sensor. The polarizer's orientation can be switched rapidly in milliseconds, for example, to 0°, 45°, 90°, and 135°. The sensor and lens are sealed within an anti-fog chamber. The environmental perception module includes a temperature and humidity sensor, installed close to the inside of the cabinet door glass, to monitor the temperature and humidity near the glass, providing prior data for condensation risk assessment. The central processing and control module is typically a high-performance embedded AI chip, responsible for running core algorithms and generating hardware control signals, such as Jetson Nano or other equivalent platforms.

[0024] The core algorithm layer, or software layer, includes image acquisition and hardware control drivers, a lightweight condensation interference assessment network, a multispectral image processing unit, a polarization image processing unit, a multimodal fusion recognition model, and a decision and system control unit. It analyzes visible light image quality in real time and outputs the degree and type of interference. The image acquisition and hardware control drivers control the light source switch, polarizer angle switching, and synchronous image acquisition. The multispectral image processing unit decodes near-infrared structured light patterns in the second imaging mode to reconstruct the 3D point cloud of the object to be identified. The polarization image processing unit performs differential calculations on multi-angle polarized images in the third imaging mode to enhance image texture. The multimodal fusion recognition model fuses different features for final classification, and the decision and system control unit executes imaging mode switching strategies based on the evaluation results. The application layer runs in the unmanned vending machine, enabling automatic product recognition and payment after opening the door, maintaining a high recognition rate even under severe condensation interference.

[0025] In this embodiment, after the system is powered on, it defaults to using the first imaging mode to acquire a first image of the item to be identified under visible light illumination. The first image contains product information and possible condensation interference. Subsequently, the system performs interference analysis on the first image. By analyzing quality characteristics such as the global brightness distribution, local texture clarity, and edge sharpness of the image, it determines the degree of interference caused by condensation, such as light, medium, or heavy levels, and identifies the distribution pattern of condensation in the imaging area as the type of interference, such as whether it is a continuous water film or discrete, isolated water droplets.

[0026] The system dynamically selects one of three imaging modes as the target imaging mode based on the degree and type of interference. It should be noted that the interference degree is divided into at least three levels: slight, significant, and severe; and the interference type is divided into at least two types: continuous distribution and discrete distribution. In cases of slight interference, the first imaging mode can continue to be used. In cases of significant interference and a continuous water film morphology, the second imaging mode is selected, utilizing the penetrability of invisible light to acquire the geometric information of the object to be identified. In cases of significant interference and a discrete water droplet morphology, the third imaging mode is selected, utilizing polarization difference to suppress specular reflection. In cases of severe interference and blurred boundaries of the interference type, the second and third imaging modes can be executed sequentially. In this embodiment, instead of passively repairing damaged images, the hardware is instructed to switch to a less interference-prone physical dimension for acquisition.

[0027] Following a defined target imaging pattern, the corresponding light source and polarizer settings are controlled to acquire a second image of the object to be identified. Target feature vectors are extracted from the second image and input into a multimodal fusion recognition model, ultimately outputting the object category. Even with severely damaged visible light images, highly robust features can still be obtained for identification.

[0028] The system defaults to using visible light illumination to acquire a first image. A lightweight neural network analyzes this first image to determine the severity and distribution pattern of the condensation. Based on the evaluation results, the most suitable anti-interference imaging mode is selected. The corresponding hardware module is controlled to acquire a second image under the selected mode. Unaffected feature vectors are extracted from the second image and input into the recognition model to obtain the product category. In this embodiment, the core role of the intelligent algorithm is shifted from data processing to perception, decision-making, and control. By actively switching to a physical perception dimension less affected by condensation, interference is avoided at the data acquisition source, thereby obtaining highly robust feature information.

[0029] In this embodiment, a first image is first acquired to determine the degree and type of interference from condensate. Then, based on the degree and type of interference, a target imaging mode is determined from a first imaging mode, a second imaging mode, and a third imaging mode. A second image is then acquired under the target imaging mode, and features are extracted for identification. Specifically, the invisible light imaging mode utilizes the strong penetrability of near-infrared light through the water film to obtain geometric features from the spectral dimension; the polarizer multi-angle imaging mode utilizes the ability of polarization difference to suppress specular reflection, obtaining enhanced texture features from the polarization dimension. These two modes avoid interference from condensate at the source of acquisition, and even when severe condensate causes significant loss of visible light image information, highly robust feature vectors can still be extracted. Determining the target imaging mode based on the degree and type of interference means that the system can identify the current degree and distribution of condensate interference in real time and adaptively select the most effective physical imaging mode accordingly. This automatic matching of the optimal image acquisition scheme for different condensate morphologies maintains stable and high-precision recognition performance in complex and changing environments, significantly improving the overall adaptability of the system.

[0030] Reference Figure 3 The diagram illustrates a flowchart of another embodiment of the item recognition method of the present invention, the method comprising: Step 201: Acquire a first image of the object to be identified, and determine the degree and type of interference caused by condensation in the first image.

[0031] Step 201 can be referred to step 101 above, and will not be repeated here.

[0032] Step 202: Input the first image into the evaluation network and output the interference level and the interference type; the evaluation network includes a first evaluation sub-network and a second evaluation sub-network, the first evaluation sub-network is used to output the interference level, and the second evaluation sub-network is used to output the interference type; the training samples of the first evaluation sub-network include multiple images acquired under various condensation conditions under visible light illumination at the first preset wavelength, and the interference level is labeled on the multiple images; the training samples of the second evaluation sub-network include multiple images acquired under various condensation conditions under visible light illumination at the first preset wavelength, and the interference type is labeled on the multiple images.

[0033] In this embodiment, the evaluation network consists of two sub-networks, which can share a shallow feature layer or be trained separately. It should be noted that the specific form of the interference level can be a global interference score, and the interference type can be distinguished by the distribution pattern of the condensate. The interference type includes at least two types: one type has a uniform condensate distribution, and the other type has a discrete condensate distribution.

[0034] Condensation water interference affects images, causing them to exhibit degradation characteristics. Different levels and types of interference result in different degradation features. Therefore, by identifying these degradation characteristics, the degree and type of interference can be inferred. When the interference type is uniform fogging or global scattering, the image exhibits a hazy, white effect with a significant decrease in contrast. Gradient changes are gentle, without sharp edges. In the frequency spectrum, low-frequency components dominate, high-frequency information is severely attenuated, and the overall image variation is relatively uniform. When the interference type is discrete water droplets, the focusing effect of the droplets on light creates irregular bright areas in the image; the droplets act like microlenses, causing local distortion, breakage, or refraction shifts at object edges; background textures are irregularly distorted through the droplets; and the interference exhibits strong spatial inhomogeneity. Therefore, when the first type of feature is identified, the interference type tends towards uniform fogging; when the second type of feature is identified, the interference type tends towards discrete distribution.

[0035] For example, the first evaluation subnetwork is a convolutional neural network with an encoder-regressor structure. It should be noted that the pre-training process of the first evaluation subnetwork includes collecting a large number of image pairs with real labels of global interference scores. The real labels can be annotated by experts or obtained by synthesizing degraded images and setting a known degree of degradation. The degraded images can be blurred with simulated fog or water droplets.

[0036] The network architecture of the first evaluation sub-network includes an encoder. Exemplarily, the encoder uses a backbone convolutional neural network, such as the convolutional part of a Residual Network (ResNet), with the final classification layer removed. The convolutional part of ResNet consists of multiple stacked convolutional layers, pooling layers, and activation functions. Shallow convolutional kernels learn low-level features such as edges and colors; deep convolutional kernels combine these low-level features to abstract higher-level semantic features, such as blocky blurred regions, overall patterns of contrast loss, and homogenized texture regions. A regression head is used for score calculation. After the feature map output by the encoder, a global average pooling layer and several fully connected layers are appended. The last fully connected layer outputs a neuron, whose value is constrained to between 0 and 1 using a sigmoid activation function; this is the predicted global interference score.

[0037] The specific training process of the first evaluation sub-network includes: using mean squared error as the loss function, and continuously adjusting all network weight parameters through backpropagation. The training goal is to make the network's output prediction score as close as possible to the true label. During training, the network learns how to map the input image to a numerical value that represents its degree of degradation. Specifically, during training, the network implicitly calculates a series of image degradation metrics, such as contrast reduction, sharpness / sharpness loss, texture entropy reduction, and specific artifact detection. Contrast reduction is due to condensation scattering light, causing the image to appear washed out and lose contrast. The algorithm in the network evaluates the global grayscale distribution and standard deviation of the image. Sharpness / sharpness loss is due to water film blurring the image; the algorithm measures the blurring degree by calculating the image's gradient magnitude, such as the response of the Laplacian operator or frequency domain features. Texture entropy reduction is due to the water film smoothing out the rich texture of the product packaging. The algorithm evaluates the complexity and richness of the local texture of the image. Specific artifact detection is due to the irregular, bright spots and contour distortions caused by discrete water droplets.

[0038] The network evaluation of the global grayscale distribution and standard deviation of the image refers to histogram statistics and moment calculation of the grayscale image. Specifically, this includes: grayscale conversion of the image, calculation of the grayscale histogram, and calculation of grayscale distribution indices. Grayscale conversion refers to converting the color image to a grayscale image I, with pixel values ​​typically ranging from 0 to 255. Calculating the grayscale histogram involves counting the number of pixels at each grayscale level i (0 ≤ i ≤ 255) in the image, resulting in a histogram array hist[i]. Calculating the grayscale distribution indices involves calculating the grayscale mean, grayscale standard deviation, and grayscale entropy. The formula for calculating the grayscale mean is: Equation (1) In equation (1), N represents the total number of pixels. It reflects the average brightness.

[0039] The formula for calculating the standard deviation of gray levels is: Equation (2) The standard deviation of grayscale is a core indicator for evaluating contrast. The smaller the σ value, the more concentrated the pixel grayscale values ​​are around the mean, resulting in lower overall image contrast and a generally hazy image appearance.

[0040] The formula for calculating grayscale entropy is: E=-Σ(p(i)*log2(p(i))) Formula (3) In equation (3), p(i) = hist[i] / N, the gray entropy value reflects the degree of disorder and information content of the gray distribution. The gray distribution of highly degraded images is concentrated and the entropy value is low.

[0041] Network computation measures blurriness using gradients or frequency domains, with spatial gradient methods being a specific implementation. This involves: convolution using the Laplacian operator, calculating the response map, and calculating the response variance. The Laplacian operator is a second-order differential operator used to highlight regions of abrupt gray-level changes in an image, which are typically edges. Calculating the response map involves convolving the gray-level image I with the Laplacian response map L. Calculating the response variance involves calculating the variance Var(L) of all pixel values ​​in the response map L. A larger variance indicates more and sharper edges in the image, resulting in a clearer image; a smaller variance indicates a more blurred image.

[0042] Network evaluation of local texture complexity and richness in an image can be achieved through Local Binary Pattern (LBP) calculation. This involves: calculating the Local Binary Pattern (LBP) map, calculating the LBP histogram, and calculating the histogram entropy. Calculating the LBP map involves taking each pixel in the grayscale image I, using its 3x3 neighborhood center pixel as a threshold, and binarizing the surrounding 8 pixels. Pixel values ​​greater than the threshold are set to 1, otherwise to 0, resulting in an 8-bit binary number, which is then converted to decimal as the LBP value for that pixel. This process is repeated across the entire image to obtain the LBP map. Calculating the LBP histogram involves statistically analyzing the frequency of all possible values ​​(0-255) in the LBP map. Calculating the histogram entropy involves calculating the entropy value E_LBP based on the LBP histogram. Regions with rich and complex textures will produce diverse LBP patterns, resulting in a more uniform histogram distribution and higher entropy values. Degraded regions have smooth textures, a single LBP pattern, a concentrated histogram, and lower entropy values.

[0043] The first evaluation subnetwork evaluates the degree of interference in the first image by calculating the aforementioned features.

[0044] The second evaluation subnetwork is a lightweight classification network, either identical to or parallel to the first evaluation subnetwork, and is trained to distinguish between different types of interference. The training data needs to include images with clearly labeled interference types. The network learns to capture the different feature patterns mentioned above: for uniform fog, the network focuses more on global statistical and frequency domain features; for discrete droplets, it focuses more on local abnormal highlights, edge distortion, and texture irregularities. Ultimately, the second evaluation subnetwork outputs a probability distribution, for example, [0.9, 0.1] indicating a 90% probability of uniform fog and a 10% probability of discrete droplets. This probabilistic form is more flexible than hard judgment and provides a confidence reference for subsequent decisions.

[0045] The goal of the second evaluation sub-network is to learn the essential differences left by the two types of interference on the image. The original RGB image or its grayscale image is input into the second evaluation sub-network. This sub-network can be designed as a feature extractor with dual attention pathways. The global context pathway uses standard convolutional layer stacking to capture the overall statistical characteristics of the image through a larger receptive field, such as overall contrast attenuation and color shift. This pathway is more sensitive to features of uniform fogging. The local detail pathway uses denser convolutions or dilated convolutions to focus on capturing abrupt changes, discontinuities, and anomalous patterns in local regions. This pathway is more sensitive to features of discrete water droplets, such as local specular highlights, edge distortion, and texture breaks. The feature maps extracted from the two pathways are fused. Subsequently, a spatial and channel attention module is used to automatically determine whether global context information or local detail anomalies are more important in the current feature set. This is equivalent to having the network determine which type of interference the current image better matches. The fused features are then converted into a feature vector through global average pooling. This feature vector is then mapped to a two-dimensional vector [z1, z2] through a fully connected layer. Finally, the probability distribution is calculated using the Softmax function.

[0046] Specifically, training the second evaluation sub-network requires constructing a dedicated dataset where each image has a clearly defined interference type label, such as uniform fog or discrete water droplets. Enhancement techniques such as rotation, scaling, and color jitter are applied to the training data, particularly synthetic data simulating different concentrations and droplet sizes, to improve the model's generalization ability.

[0047] The second evaluation sub-network automatically learns features to distinguish between two types of interference through its various convolutional layers. The bottom convolutional layers, or the first convolutional kernel, learn Gabor-like primitives after training, sensitive to edges, spots, and textures in specific directions. For water droplet images, these filters produce strong activations at water droplet boundaries and in bright spot regions; for foggy images, the activation responses are generally weaker and more scattered. Middle or deeper convolutional layers combine the simple features from the bottom layers to form more complex pattern detectors. The middle convolutional layers are used to learn to identify globally low-contrast uniform regions and locally bright distortion clusters. For example, some feature maps respond strongly to large-area smooth grayscale changes, while others respond strongly to dense, small-scale high-gradient clusters. After the convolutional layers, a global average pooling layer compresses each feature map into a scalar, essentially summarizing spatial information into intensity information.

[0048] The last fully connected layer of the network has two neurons corresponding to uniform atomization and discrete droplets, respectively. Its output is called logits, which are the raw scores. The Softmax activation function transforms the two raw scores [z_atomization, z_droplet] into a probability distribution. The calculation formula is: P_atomization = exp(z_atomization) / (exp(z_atomization) + exp(z_droplet)) P_waterdrop = exp(z_waterdrop) / (exp(z_atomization) + exp(z_waterdrop)) The values ​​of P_fog and P_droplet are both between 0 and 1, and their sum equals 1. This gives us the probability distribution of the two types of interference affecting the image.

[0049] The training process approximates the probability distribution to the true labels, using cross-entropy loss as the loss function. For an image labeled "foggy" (i.e., image with labels [1,0]), the loss function is calculated as: Loss = -(1*log(P_foggy) + 0*log(P_droplet)) = -log(P_foggy). The training objective is to minimize this loss. This means the network adjusts all its parameters, such as convolutional kernel weights, to make the output P_foggy as close to 1 as possible for foggy images and P_droplet as close to 1 as possible for droplet images. This optimization is achieved iteratively through backpropagation and gradient descent algorithms.

[0050] When deploying the second evaluation network, an image is input, propagated forward through the network, and the final two-dimensional vector output by the Softmax layer represents the probability distribution of the interference type. If P_fog > 0.8, the system can determine that the main interference type is uniform fogging. If P_droplets > 0.7, the system can determine that the main interference is droplets, triggering different processing steps or indicating low reliability in identifying the image region. If the probability distribution is close to [0.5, 0.5], it indicates low confidence in the judgment and a relatively ambiguous distinction between the two interference types.

[0051] In other words, a binary classification CNN with a Softmax output layer is trained as the second evaluation sub-network. Through its internal hierarchical structure, the second evaluation sub-network automatically learns the visual features that distinguish between fog and water droplets from the data; and the combination of the Softmax function and the cross-entropy loss function ensures that the second evaluation sub-network can output a probability distribution that can be used to quantify confidence.

[0052] In this embodiment, two sub-networks are trained by constructing a dataset labeled with global interference scores and interference type labels. By fusing implicit deep features with explicit image degradation indicators, the network can automatically learn the essential characteristics of condensation interference without the need for manually designing complex image quality assessment rules. The dual-path attention structure makes the network more accurate in distinguishing between the two interference forms.

[0053] Step 203: If the interference level is less than the first preset threshold, determine the target imaging mode as the first imaging mode; Step 204: If the interference level is greater than or equal to the first preset threshold and less than the second preset threshold, determine the target imaging mode as the second imaging mode or the third imaging mode according to the interference type; the first preset threshold is less than the second preset threshold. Step 205: If the interference level is greater than or equal to the second preset threshold, or if the difference between the probability of the interference type being the first type and the probability of the interference type being the second type is within a preset range, determine the target imaging mode as the second imaging mode and execute the third imaging mode in sequence; the distribution pattern of the condensate in the second type is more discrete than the distribution pattern of the condensate in the first type.

[0054] Regarding steps 203 to 205, it should be noted that the degree of interference is evaluated by assessing the global interference score output by the network. The global interference score is a continuous value from 0 to 1. The global interference score is used to quantify the severity of the degradation of visible light images by condensation. The smaller the global interference score value, the clearer the image; the larger the global interference score value, the more severe the interference to the image.

[0055] In this embodiment, the interference level is divided into three levels: mild interference, moderate interference, and severe interference. The first preset threshold T1 is a critical value used to distinguish between mild and moderate interference; interference below T1 is considered acceptable, while interference above T1 requires intervention. The second preset threshold is a critical value used to distinguish between moderate and severe interference, where T2 > T1; interference above T2 is considered severe interference.

[0056] The first type refers to condensate water exhibiting a continuous and uniform distribution, i.e., uniform atomization / water film. Image characteristics affected by this type of interference include: decreased overall contrast, global blurring, absence of local highlight distortion, low-frequency dominance in the frequency domain, and smooth spatial variation. The second type refers to condensate water exhibiting a discrete and non-uniform distribution, i.e., discrete water droplets. Image characteristics affected by this type of interference include: the presence of local irregular bright spots, refractional shift or breakage at object edges, texture distortion, and uneven spatial distribution.

[0057] The first imaging mode refers to the mode of collecting images under visible light irradiation, and is also the initial default mode. The second imaging mode refers to the mode of collecting images under invisible light irradiation, which is used to counter uniform atomization-type interference. The third imaging mode refers to the mode of collecting images with the polarizer arranged at at least two different angles, which is used to counter discrete water droplet-type interference.

[0058] The decision is divided into three scenarios according to the interference degree and interference type. When the global interference score is less than T1, it indicates that the image is slightly interfered, the visible light image is still reliable, there is no need to switch the program mode, and the first imaging mode is continuously adopted. When T1 ≤ the global interference score < T2, it indicates that the image is moderately interfered, and the most effective physical dimension needs to be selected according to the interference type; when the global interference score ≥ T2, or the probability difference between the two types is within the preset range, that is, in the case of ambiguous judgment, the second imaging mode and the third imaging mode are executed in sequence. This is because when the interference is severe or the type is unclear, a single mode may be insufficient, so the two anti-interference modes are used sequentially to obtain complementary features.

[0059] In the embodiments of the present application, when the first image is slightly interfered, it indicates that the quality of the visible light image is still acceptable, and the condensed water has limited impact on recognition at this time. If other modes are switched to rashly, the system overhead and delay will be increased instead. Therefore, maintaining the first imaging mode and directly using the visible light image for recognition has the highest efficiency. When the first image is moderately interfered, the visible light image has been significantly damaged, but has not yet reached the degree of complete failure. At this time, targeted countermeasures need to be taken according to the specific form of the condensed water. When the first image is severely interfered or the interference type judgment is ambiguous, when the interference is extremely serious, any single mode may not be able to extract sufficiently reliable complete features; when the probabilities of the two interference types are close, for example, the probability distribution of the two interference types is [0.5, 0.5], the system cannot be sure whether the condensed water is atomized or in the form of water droplets. At this time, the two modes are executed sequentially to obtain three-dimensional geometric features and enhanced texture features at the same time, and then adaptive weighted fusion is performed through the subsequent multi-modal fusion recognition model. Through the complementarity of the dual modes, the recognition robustness under extremely complex conditions is ensured to the greatest extent.

[0060] In this embodiment, by setting a first preset threshold and a second preset threshold, and introducing interference type probability difference judgment, the decision is finely divided into three scenarios: mild interference retains the visible light mode to maintain low power consumption and high efficiency; moderate interference switches to the corresponding anti-interference mode according to the interference pattern; and when there is severe interference or ambiguity in the type judgment, two anti-interference modes are executed sequentially to obtain complementary features. A hierarchical and progressive adaptive decision framework is constructed to avoid the problem of insufficient adaptability of a single fixed strategy when facing dynamic changes in condensate morphology. By defining the interference type as the discreteness of the condensate distribution pattern, the decision is directly based on the objective differences in physical morphology, rather than a vague image degradation description, thereby accurately matching the most effective physical imaging dimension and maintaining a high recognition rate even in complex scenarios where fog and water droplets coexist.

[0061] Optionally, step 204 may specifically include the following sub-steps: Sub-step 2041: If the interference type is the first type, determine the target imaging mode as the second imaging mode; Sub-step 2042: If the interference type is the second type, determine the target imaging mode as the third imaging mode.

[0062] For sub-steps 2041 and 2042, the first type refers to the distribution of condensate as a continuous and uniform distribution; the second type refers to the distribution of condensate as a discrete and non-uniform distribution, i.e., discrete water droplets.

[0063] When the interference type is determined to be uniform fogging, the visible light image becomes blurred overall due to uniform fogging. However, invisible light has strong penetrability through the water film, allowing it to bypass scattering and obtain the three-dimensional geometric features of the product, thus fundamentally avoiding interference. Therefore, in the case of the first type of interference, the second imaging mode is adopted, acquiring images under invisible light illumination.

[0064] When the interference type is determined to be discrete water droplets, the strong specular reflection and refraction caused by these droplets lead to localized highlights and distortion. By acquiring images at different angles using a polarizer and calculating the difference, reflected light can be effectively suppressed, and obscured texture details can be restored. Therefore, when the interference type is type two, a third program mode is adopted, which involves acquiring images at different angles by switching polarizers.

[0065] In this embodiment, a precise adversarial mapping is designed based on the physical and optical properties of the two types of interference. The essence of uniform fogging is light scattering, which leads to texture blurring. Near-infrared structured light avoids the influence of scattering by obtaining geometric features through penetrating the water film. The essence of discrete water droplets is specular reflection and refraction, which leads to local highlights and distortion. Polarization difference restores the texture by suppressing reflected light. This maximizes the unique advantages of each physical method and significantly improves the robustness of feature extraction.

[0066] Step 206: When the target image mode is the second imaging mode, turn off the visible light, project the second preset wavelength invisible light onto the surface of the object to be identified in a preset pattern, and acquire the second image of the object to be identified; Step 207: Based on the second image, construct a 3D point cloud of the object to be identified; Step 208: Extract shape feature vectors from the three-dimensional point cloud, and determine the item category of the item to be identified based on the shape feature vectors.

[0067] For steps 206 to 208, the second imaging mode refers to the mode of acquiring images of the item to be identified under invisible light illumination, specifically a near-infrared structured light three-dimensional imaging mode. The second preset wavelength has strong penetration through water films and is invisible to the human eye, thus not interfering with customers. For example, in this embodiment, 940nm near-infrared light is used. The preset pattern refers to an optical pattern with specific spatial encoding, such as sinusoidal fringes or random speckle. The second image is an image sequence acquired under the second imaging mode, containing a structured light pattern modulated by the surface topography of the product. The three-dimensional point cloud is a collection of numerous three-dimensional spatial points, describing the geometry of the product surface. The shape feature vector refers to the quantized geometric parameters extracted from the point cloud, such as height, volume, minimum circumscribed cuboid size, surface curvature statistics, etc.

[0068] The recognition model refers to a multimodal fusion recognition model that can accept shape feature vectors or texture feature vectors and output the item category. When a single type of feature vector is input, the model's internal attention mechanism automatically focuses on that feature.

[0069] Specifically, in the second imaging mode, the visible light LED is turned off to avoid interference. The near-infrared structured light projector is turned on to project a preset coded pattern onto the surface of the object to be identified. The polarizer is either set to a fixed orientation or removed to ensure smooth passage of near-infrared light. The global shutter CMOS sensor synchronously acquires an image sequence modulated by the object's surface morphology. If phase-shifting fringes are used, typically 3... Four fringe patterns with different phases were used to acquire corresponding image sequences. If random speckle patterns were used, a single frame of speckle pattern was projected, and a deformed speckle pattern was acquired. Near-infrared light has strong penetrability to water films, and its three-dimensional shape is not sensitive to texture distortion caused by discrete water droplets; therefore, the acquired patterns are minimally affected by condensation.

[0070] In the second imaging mode, the acquired image sequence is processed, and the three-dimensional geometric information of the product surface is recovered from the two-dimensional image using the principle of active stereo vision. If phase-shifting fringe light is projected, three or four phase-shifting patterns are typically required. For each pixel in the image, the absolute phase value corresponding to that point is calculated using inverse trigonometric functions based on its light intensity value in different phase-shifting patterns. This phase value corresponds one-to-one with the angle or position of the projector's outgoing light. If a random speckle pattern is projected, a reference speckle pattern is pre-stored or its digital template is obtained through calibration. By performing rapid local window matching within a large field of view, the position most similar to each small region in the reference pattern is found in the acquired image, thus establishing the correspondence between image points and projector pattern points. After successfully establishing the correspondence between each pixel in the image and the projector pattern, three-dimensional reconstruction is performed using the intrinsic and extrinsic parameters obtained through high-precision calibration. The calibration information includes: the camera's intrinsic parameters, such as focal length, principal point, and distortion coefficient; the projector's intrinsic parameters; and the relative position and orientation between the camera and the projector, where the projector can be considered equivalent to a virtual camera.

[0071] For each pixel, it and its corresponding projector spot form a ray of light in space. According to the principle of triangulation, the line connecting this pixel to the camera's optical center and the line connecting the corresponding projector spot to the projector center intersect in space. By solving for the intersection point or the least squares solution of these two rays, the three-dimensional spatial coordinates of the object's surface point corresponding to that pixel can be calculated. By traversing all pixels and collecting all the calculated three-dimensional points, a three-dimensional point cloud covering the visible surface of the product is formed.

[0072] Geometric feature vectors F_3D, including height, volume, minimum bounding box size, and surface curvature, are extracted from the point cloud. These feature vectors are then input into the multimodal fusion recognition model. During training, the multimodal fusion recognition model has already learned the mapping from geometric features to product SKUs. When only a single feature is input, the model's internal attention mechanism degenerates, automatically concentrating all weights on that feature without cross-modal allocation. The model outputs the item category and recognition confidence score.

[0073] In this embodiment, visible light is turned off in the second imaging mode, and a near-infrared structured light projector is turned on to project the coded pattern onto the product surface. The pattern modulated by the surface morphology is acquired, and a three-dimensional point cloud is reconstructed through stereo matching or phase calculation. Geometric features such as height, volume, size, and curvature are extracted from it. Utilizing the strong penetrability of near-infrared light to water films, geometric information is directly obtained from the spectral dimension. Even if uniform fogging causes the visible light image to be blurred as a whole and the texture to be completely lost, the three-dimensional shape features can still be accurately reconstructed, and the geometric features are not sensitive to local texture distortion caused by discrete water droplets. The influence of water film scattering is fundamentally avoided, and the extracted geometric features have natural robustness, thus enabling reliable recognition even in heavily fogged scenarios.

[0074] Step 209: When the target imaging mode is the third imaging mode, set the polarizer to the first direction and acquire the first candidate image of the object to be identified; Step 210: Set the polarizer to the second direction and acquire a second candidate image of the object to be identified; the first direction and the second direction are orthogonal. Step 211: Calculate the difference image between the first candidate image and the second candidate image, and determine the difference image as the second image; Step 212: Extract texture feature vectors from the second image and determine the item category of the item to be identified based on the texture feature vectors.

[0075] For steps 209 to 212, the third imaging mode refers to a mode in which the polarizer is set at at least two different angles and images are acquired. The mode suppresses specular reflection and enhances texture by calculating the differential image; this is known as active polarization differential imaging mode. The polarizer is an electrically controlled liquid crystal polarizer placed in front of the image sensor. Driven by an electrical signal, it can rapidly switch the polarization direction, for example, within milliseconds, to 0°, 45°, 90°, 135°, etc. The first and second directions are two orthogonal polarization directions, such as 0° and 90°, which are perpendicular to each other.

[0076] First candidate image Second candidate image These are images acquired in the first and second directions, respectively. Differential imaging is achieved using the formula PD = -k* The calculated image, where k is a coefficient adaptively calculated based on scene brightness, effectively suppresses unpolarized scattered light from the condensate surface and enhances polarized reflection details on product packaging, such as trademarks and patterns.

[0077] Texture feature vectors are features extracted from differential images that describe surface texture information, such as SIFT features or deep learning features. The recognition model is a multimodal fusion recognition model that automatically focuses on this feature when given a single modality input, and outputs the item category.

[0078] For example, in the third imaging mode, visible light illumination or an optional near-infrared uniform light source is maintained to ensure that polarization difference is effective in the visible light band. First, the liquid crystal polarizer is driven to a first direction, such as 0°, and a first candidate image is acquired. Within an extremely short time, such as milliseconds, the polarizer is switched to a second direction orthogonal to the first direction, for example, 90°, and a second candidate image is acquired simultaneously. The two data acquisitions need to be completed quickly and continuously to ensure scene consistency and real-time performance. The specular reflection light generated by the surface of discrete water droplets from condensation is strongly polarized, while the diffuse reflection light from the product packaging is unpolarized. Under orthogonal polarization directions, the intensity difference of the specular reflection light is significant, while the intensity change of the unpolarized light is smaller in both directions. Therefore, subsequent differential operations can effectively suppress specular reflection and highlight the texture details of diffuse reflection.

[0079] Calculate the polarization difference image PD=I_ -k* , It is an image acquired from the first direction, containing strong specular reflection components and diffuse textures; The image is acquired in an orthogonal direction, with specular reflection components greatly suppressed, mainly consisting of diffuse textures and a small amount of ambient light. k is an adaptive coefficient, dynamically calculated based on scene brightness, to make the uniform background response in the difference result approach zero, highlighting polarization-preserved textures.

[0080] Texture feature vectors are extracted from the PD image, such as Scale-Invariant Feature Transform (SIFT) features, or other texture descriptors. The feature vector is denoted as F-PD and is used to represent the detailed information of the product surface. The texture feature vector F-PD is input into the multimodal fusion recognition model. Since it is only a single modality at this point, the model's internal attention mechanism is in a degenerate state. The multimodal fusion recognition model only processes the currently input texture feature, automatically concentrating all weights on this feature. There is no cross-modal weight allocation. The model outputs the item category and recognition confidence score.

[0081] In this embodiment, visible light illumination is maintained in the third imaging mode. The liquid crystal polarizer is quickly switched to two orthogonal directions, and two images are acquired simultaneously. A polarization difference image is calculated, and texture features are extracted from the difference image. Utilizing the strong polarization characteristics of specular reflected light and the difference between it and unpolarized diffuse reflected light, specular highlights generated by discrete water droplets are physically suppressed through differential operations. The adaptive coefficient k ensures the best suppression effect under different brightness scenes, allowing texture details such as trademarks, patterns, and text that were originally obscured by highlights to be clearly presented. The reflection component is directly separated at the physical level, without relying on the repair of damaged data. Therefore, the suppression of discrete water droplet interference is more thorough and stable.

[0082] Step 213: In the second imaging mode, acquire a third candidate image of the object to be identified and determine the corresponding first feature vector based on the third candidate image; Step 214: In the third imaging mode, acquire a fourth candidate image of the object to be identified and determine the corresponding second feature vector based on the fourth candidate image; Step 215: Determine the item category of the item to be identified based on the first feature vector and the second feature vector.

[0083] For steps 213 to 215, when the interference level is extremely high or the interference type is ambiguous, the system decides to execute the second imaging mode and the third imaging mode sequentially. First, the second imaging mode is executed to obtain the first feature vector, which is the shape and geometric features extracted from the 3D point cloud. Then, the third imaging mode is executed to obtain the second feature vector, which is the texture features extracted from the polarization difference image. The two feature vectors describe the attributes of the same object to be identified from different physical dimensions. Finally, these two feature vectors are used together to determine the object category.

[0084] For example, in the second imaging mode, visible light is turned off, and a near-infrared structured light projector is turned on to project a coded pattern onto the product surface; a pattern sequence modulated by the product surface morphology is acquired; a three-dimensional point cloud is reconstructed using stereo matching or phase resolution algorithms; geometric parameters are extracted from the point cloud to form a first feature vector. Because near-infrared light penetrates the water film in the second imaging mode, the three-dimensional shape is insensitive to texture distortion caused by condensation.

[0085] In the third imaging mode, while maintaining visible light illumination, the liquid crystal polarizer is sequentially switched to two orthogonal directions, such as 0° and 90°, to simultaneously acquire two images; the polarization difference image PD is calculated. -k* Texture descriptors are extracted from the difference image to form a second feature vector. A third imaging mode is used to suppress specular highlights from discrete water droplets, enhancing texture details such as occluded trademarks and patterns.

[0086] The first and second feature vectors are used together as input to the recognition model. The recognition model is a pre-trained classifier that receives one or more feature vectors and outputs the corresponding item category and recognition confidence score. The two feature vectors can be concatenated, superimposed, or combined in other ways before being input to the model, or they can be directly used as two independent input channels. During the training phase, the recognition model has already learned the mapping relationship from shape features and texture features to various types of goods. When the model receives two feature vectors, it combines the information from both based on the parameters obtained during training to output the final item category.

[0087] In this embodiment, when interference is severe or type judgment is ambiguous, the second and third imaging modes are executed sequentially to obtain shape feature vectors and texture feature vectors respectively, and both are used as the target feature vector. Through complementary acquisition of two physical dimensions, a more comprehensive description of product information is constructed. Sequential execution of the dual modes organically combines geometric and texture information, providing a richer feature foundation for subsequent fusion recognition.

[0088] Optionally, step 215 may specifically include the following sub-steps: Step 2151: Based on the confidence scores and historical recognition results of the first feature vector and the second feature vector in the current environment, respectively, assign corresponding fusion weights to the first feature vector and the second feature vector; Step 2152: Perform weighted fusion of the first feature vector and the second feature vector according to the fusion weight to obtain a fused feature vector; Step 2153: Determine the item category of the item to be identified based on the fused feature vector.

[0089] For steps 2151 to 2153, the first feature vector refers to the shape feature vector F-3D extracted under the second imaging mode, describing the three-dimensional geometric information of the product. The second feature vector refers to the texture feature vector F-PD extracted under the third imaging mode, describing the polarization-enhanced texture of the product surface. Confidence refers to the reliability of each feature mode under the current interference environment. For example, geometric features are more reliable under heavy fog; texture features are more reliable under discrete water droplets. Historical recognition results refer to the short-term memory maintained by the multimodal fusion recognition model, encoding the mode state, feature weights, and recognition confidence of several past recognition cycles. Fusion weights are coefficients assigned to the first and second feature vectors, used for weighted summation, reflecting the importance of each feature vector to the current recognition result. The fusion feature vector is a comprehensive feature representation obtained after weighted summation, fusing complementary information from two physical dimensions. The classifier is the last layer of the recognition model, such as a fully connected layer and Softmax, used to output the item category and confidence.

[0090] Based on the reliability of features in the current environment and historical recognition experience, the fusion weights of the two feature vectors are dynamically calculated instead of using fixed weights. For example, in extremely dense fog, the reliability of the shape feature vector F-3D is higher than that of the texture feature vector F-PD; under discrete water droplet interference, the texture feature vector F-PD is more reliable; for goods with regular shapes but rich textures, F-PD has stronger discriminative power; for goods with unique shapes but highly reflective packaging, F-3D has stronger discriminative power. This information is implicit in the numerical distribution of the feature vectors themselves.

[0091] It should be noted that the input to the multimodal fusion recognition model includes a mode state vector M, a history context vector H, and features F to be fused. The mode state vector is a multidimensional vector, for example, M=[m1,m2,s1,s2]. Here, m1=1,m2=0 represents the second imaging mode, and m1=0,m2=1 represents the third imaging mode. s1 and s2 are the uniform fog score and discrete water droplet score output by the interference evaluation network when the mode is triggered. The history context vector H is a fixed-dimensional vector maintained and updated by a recurrent neural network unit such as LSTM. The history context vector encodes key information from the past several recognition cycles; for example, H=LSTM_State([previous mode M, previous feature weights, previous recognition confidence]). The features F to be fused are obtained by concatenating or projecting the shape feature vector F-3D and the texture feature vector F-PD into the same dimensional space, resulting in F=[F-3D;F-PD].

[0092] By concatenating vectors M and H and passing them through a small neural network, a set of dynamic attention key vectors is generated. and a summary query vector . It is not generated from feature F itself, but determined by the current system state. Dynamically generated... and Attention is calculated. Simultaneously, the original features F are projected into standard key vectors. The sum of the value vectors V. The final attention weights α are obtained through... At the same time with and After interaction, the formula for calculating attention weights is as follows: )+ Formula (4) In equation (4), λ1 and λ2 are learnable parameters used to balance the influence of prior system knowledge from patterns and history with the influence of current data from the features themselves. Weighted fusion is the weighted summation of the value vector V using calculated weights α to obtain the final context-aware fused features. The generation of weights α explicitly depends on the input pattern vector M and history vector H. The model learns to interpret M and H during training. For example, when M represents the second imaging pattern and the history H shows that the recent shape feature vector is reliable, the generated... Will guide Pay more attention to the feature channels corresponding to F-3D, and thus assign them higher weights.

[0093] In this embodiment, after obtaining two feature vectors, fusion weights are dynamically assigned based on their confidence levels in the current environment and historical recognition results. The fused feature vector is obtained through weighted summation and then input into the classifier to output the item category, thus achieving adaptive feature fusion based on an attention mechanism. The dynamic weights can highlight the most reliable features according to the actual scene. Geometric feature weights automatically increase in dense fog environments, and texture feature weights automatically increase in discrete water droplet environments. Simultaneously, historical information, such as the recognition results of previous frames and feature stability, ensures the continuity of weight allocation, avoiding misjudgments caused by single-frame anomalies. This intelligent fusion method enables the system to autonomously select the most effective information channel under complex and variable condensation water interference, thereby achieving higher recognition accuracy and confidence than a single mode.

[0094] Optionally, the method further includes: Step S11: Determine the confidence level of this identification process; Step S12: If the confidence level is greater than or equal to the preset success threshold, determine that the identification process is complete; Step S13: If the confidence level is less than the preset success threshold, re-acquire the first image of the item to be identified or output a prompt message. After receiving the operation in response to the prompt message, re-acquire the first image of the item to be identified.

[0095] For steps S11 to S13, confidence level refers to the degree of certainty the multimodal fusion recognition model has regarding the item category of its output. It is typically a value between 0 and 1, such as the maximum value output by the Softmax layer. A higher value indicates that the model is more confident in the correctness of the recognition result. The preset success threshold is a confidence threshold preset by the system, for example, 0.85. A confidence level higher than this threshold indicates that the recognition result is reliable; a confidence level lower than this threshold indicates that the recognition may be erroneous.

[0096] Re-capturing the first image refers to the system automatically or through user interaction restarting the entire recognition process from the initial steps when the recognition confidence is insufficient. Prompt messages are given to the user via the vending machine screen or voice instructions, such as "Please wait a moment" or "Please slightly adjust the product's position," guiding the user to cooperate with the re-recognition process.

[0097] If the confidence level is less than the preset success threshold, the current recognition process is considered a low-confidence sample. A low-confidence sample refers to the data record saved when recognition fails, including the original image, the imaging mode used, and the extracted feature vectors, which are used for subsequent offline model optimization. When the confidence level is greater than or equal to the preset success threshold, the system considers the recognition result reliable. Upon receiving the recognition result, the system reports the item category to the vending machine control system for deduction of payment, inventory recording, etc. The system returns to normal monitoring status, continues to acquire the first visible light image, assesses interference, and prepares for the next recognition attempt.

[0098] In this embodiment, when the recognition confidence level is lower than a preset success threshold, low-confidence samples are saved for offline model updates, and re-collection is automatically triggered or triggered after user interaction. This constructs a complete intelligent closed loop from perception, decision-making, execution to evaluation and learning. By saving failure cases, the system can continuously accumulate challenging scenarios encountered in actual deployment and use them for subsequent model retraining or fine-tuning, making the system more accurate with use and possessing self-evolution capabilities. Simultaneously, the combination of automatic retries and user guidance minimizes the impact of temporary interference on user experience and avoids transaction interruptions due to a single recognition failure. The closed-loop optimization mechanism significantly improves the system's long-term stability and environmental adaptability, and reduces manual maintenance costs.

[0099] In summary, the object-oriented method provided in this application shifts the core role of intelligent algorithms from data processing to perception, decision-making, and control. By evaluating the network in real-time analysis of the degree and form of condensation interference, the system actively switches to invisible light structured light imaging or polarization differential imaging to acquire original signals that are less susceptible to interference from different physical dimensions. Even when heavy condensation causes irreversible loss of visible light image information, highly robust features can still be acquired. Furthermore, condensation interference is refined into three scenarios: mild, moderate, and severe / blurred, corresponding to retaining the visible light mode, a targeted single anti-interference mode, and sequential execution of dual modes, respectively. The system can dynamically select the most effective countermeasures based on the actual distribution of condensation. This adaptive capability enables stable and high-precision recognition performance under complex and variable condensation conditions, significantly improving the overall adaptability and practicality of the system in high-humidity commercial environments. When executing dual modes sequentially, based on feature confidence and historical recognition status, the fusion weights of geometric and texture features are dynamically allocated through an attention mechanism, rather than simple splicing or fixed weighting. It can adaptively highlight the most reliable features based on the current environment and the discriminative characteristics of the product itself. The intelligent fusion approach enables the system to obtain high-confidence comprehensive recognition results even under extreme conditions of severe interference or type ambiguity, exhibiting stronger robustness than any single mode. By saving low-confidence samples and updating the evaluation network or recognition model offline, combined with automatic or interactive re-collection, a complete intelligent closed loop is formed, from perception, decision-making, execution to evaluation and learning. The system continuously accumulates data from challenging scenarios in practical use and uses this data for model retraining, giving the system self-evolution capabilities.

[0100] Device Examples Reference Figure 4 The diagram illustrates a logic block diagram of an item identification device according to an embodiment of the present invention. The device may include: Image evaluation module 301 is used to acquire a first image of the object to be identified and determine the degree and type of interference caused by condensation in the first image; The imaging mode determination module 302 is used to determine a target imaging mode from a first imaging mode, a second imaging mode, and a third imaging mode according to the interference level and the interference type; the first imaging mode is a mode for acquiring images under visible light illumination, the second imaging mode is a mode for acquiring images under invisible light illumination, and the third imaging mode is a mode for acquiring images with the polarizer set at at least two different angles. The category determination module 303 is used to acquire a second image of the object to be identified in the target imaging mode, and determine the category of the object to be identified based on the target feature vector corresponding to the second image.

[0101] Optionally, the imaging mode determination module 302 includes: The first imaging mode determination submodule is used to determine the target imaging mode as the first imaging mode when the interference level is less than the first preset threshold. The second imaging mode determination submodule is used to determine the target imaging mode as a second imaging mode or a third imaging mode according to the type of interference when the interference level is greater than or equal to a first preset threshold and less than a second preset threshold; the first preset threshold is less than the second preset threshold. The third imaging mode determination submodule is used to determine the target imaging mode as the second imaging mode and the third imaging mode in sequence when the interference level is greater than or equal to the second preset threshold, or when the difference between the probability of the interference type being the first type and the probability of the interference type being the second type is within a preset range; the distribution pattern of the condensate in the second type is more discrete than the distribution pattern of the condensate in the first type.

[0102] Optionally, the second imaging mode determination submodule includes: The first imaging mode determination unit is configured to determine the target imaging mode as the second imaging mode when the interference type is the first type. The second imaging mode determination unit is used to determine the target imaging mode as the third imaging mode when the interference type is the second type.

[0103] Optionally, the target feature vector includes a shape feature vector; the category determination module 303 includes: The first image acquisition submodule is used to turn off the visible light when the target image mode is the second imaging mode, project the invisible light of the second preset wavelength onto the surface of the object to be identified in the form of a preset pattern, and acquire the second image of the object to be identified. A point cloud construction module is used to construct a three-dimensional point cloud of the object to be identified based on the second image; The first category determination submodule is used to extract shape feature vectors from the three-dimensional point cloud and determine the item category of the item to be identified based on the shape feature vectors.

[0104] Optionally, the target feature vector includes a texture feature vector; the category determination module 303 includes: The second image acquisition submodule is used to set the polarizer to the first direction and acquire the first candidate image of the object to be identified when the target imaging mode is the third imaging mode. The third image acquisition submodule is used to set the polarizer to the second direction and acquire a second candidate image of the object to be identified; the first direction and the second direction are orthogonal to each other. The difference image calculation module is used to calculate the difference image between the first candidate image and the second candidate image, and determine the difference image as the second image; The second category determination submodule is used to extract texture feature vectors from the second image and determine the item category of the item to be identified based on the texture feature vectors.

[0105] Optionally, the target feature vector includes a first feature vector and a second feature vector; the category determination module 303 includes: The fourth image acquisition submodule is used to acquire a third candidate image of the object to be identified in the second imaging mode and determine the corresponding first feature vector based on the third candidate image; The fifth image acquisition submodule is used to acquire a fourth candidate image of the object to be identified in the third imaging mode and determine the corresponding second feature vector based on the fourth candidate image; The third category determination submodule is used to determine the item category of the item to be identified based on the first feature vector and the second feature vector.

[0106] Optionally, the third category determination submodule includes: The weight determination module is used to assign corresponding fusion weights to the first feature vector and the second feature vector based on the confidence level and historical recognition results of the first feature vector and the second feature vector in the current environment, respectively. The feature fusion module is used to perform weighted fusion of the first feature vector and the second feature vector according to the fusion weight to obtain a fused feature vector; The item category output submodule is used to determine the item category of the item to be identified based on the fused feature vector.

[0107] Optionally, the image evaluation module 301 includes: An image input module is used to input the first image into an evaluation network and output the degree of interference and the type of interference. The evaluation network includes a first evaluation sub-network and a second evaluation sub-network. The first evaluation sub-network is used to output the degree of interference, and the second evaluation sub-network is used to output the type of interference. The training samples of the first evaluation sub-network include multiple images acquired under various condensation conditions under visible light illumination at the first preset wavelength, and the multiple images are labeled with the degree of interference. The training samples of the second evaluation sub-network include multiple images acquired under various condensation conditions under visible light illumination at the first preset wavelength, and the multiple images are labeled with the type of interference.

[0108] Optionally, the device further includes: The confidence level determination module is used to determine the confidence level of this identification process; The identification completion module is used to determine that the identification process is complete when the confidence level is greater than or equal to a preset success threshold. The image re-acquisition module is used to re-acquire the first image of the object to be identified or output a prompt message when the confidence level is less than the preset success threshold, and to re-acquire the first image of the object to be identified after receiving an operation in response to the prompt message.

[0109] In summary, the object device provided in this application shifts the core role of intelligent algorithms from data processing to perception, decision-making, and control. By evaluating the network in real-time analysis of the degree and form of condensation interference, the system actively switches to invisible light structured light imaging or polarization differential imaging to acquire original signals that are not easily affected by interference from different physical dimensions. Even when heavy condensation causes irreversible loss of visible light image information, highly robust features can still be collected. Furthermore, condensation interference is refined into three scenarios: mild, moderate, and severe / blurred, corresponding to retaining the visible light mode, a targeted single anti-interference mode, and sequential execution of dual modes, respectively. The system can dynamically select the most effective countermeasures based on the actual distribution of condensation. This adaptive capability enables stable and high-precision recognition performance under complex and variable condensation conditions, significantly improving the overall adaptability and practicality of the system in high-humidity commercial environments. When executing dual modes sequentially, based on feature confidence and historical recognition status, the fusion weights of geometric and texture features are dynamically allocated through an attention mechanism, rather than simple splicing or fixed weighting. It can adaptively highlight the most reliable features based on the current environment and the discriminative characteristics of the product itself. The intelligent fusion approach enables the system to obtain high-confidence comprehensive recognition results even under extreme conditions of severe interference or type ambiguity, exhibiting stronger robustness than any single mode. By saving low-confidence samples and updating the evaluation network or recognition model offline, combined with automatic or interactive re-collection, a complete intelligent closed loop is formed, from perception, decision-making, execution to evaluation and learning. The system continuously accumulates data from challenging scenarios in practical use and uses this data for model retraining, giving the system self-evolution capabilities.

[0110] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0111] Reference Figure 5 This is a structural block diagram of an electronic device for item identification provided in an embodiment of this application. Figure 5 As shown, the electronic device includes: a processor, a memory, a communication interface, and a communication bus. The processor, the memory, and the communication interface communicate with each other through the communication bus. The memory is used to store executable instructions, which cause the processor to execute the method of the aforementioned embodiment.

[0112] The processor can be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable devices, transistor logic devices, hardware components, or any combination thereof. The processor can also be a combination that implements computational functions, such as a combination of one or more microprocessors, or a combination of a DSP and a microprocessor.

[0113] The communication bus may include a path for transmitting information between the memory and the communication interface. The communication bus may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 The symbol is represented by only one line, but this does not mean that there is only one bus or one type of bus.

[0114] The memory may be ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or it may be EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory), magnetic tape, floppy disk, and optical data storage devices, etc.

[0115] This application also provides a non-transitory computer-readable storage medium that, when the instructions in the storage medium are executed by a processor of an electronic device (server or terminal), enables the processor to perform the methods of the foregoing embodiments.

[0116] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above embodiments and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0117] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0118] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0119] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, embodiments of this application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of this application can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0120] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0121] These computer program instructions may also be stored in a computer-readable storage medium capable of directing a computer or other programmable data processing terminal device to operate in a predictive manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0122] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0123] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0124] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0125] The above provides a detailed description of the article identification method, apparatus, electronic device, and readable storage medium provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. An item recognition method, characterized in that, The method includes: Acquire a first image of the object to be identified, and determine the degree and type of interference caused by condensation in the first image; Based on the degree and type of interference, a target imaging mode is determined from a first imaging mode, a second imaging mode, and a third imaging mode; the first imaging mode is a mode for acquiring images under visible light illumination, the second imaging mode is a mode for acquiring images under invisible light illumination, and the third imaging mode is a mode for acquiring images with the polarizer set at at least two different angles. A second image of the object to be identified is acquired in the target imaging mode, and the object category of the object to be identified is determined based on the target feature vector corresponding to the second image.

2. The method according to claim 1, characterized in that, The step of determining the target imaging mode from the first imaging mode, the second imaging mode, and the third imaging mode based on the interference level and the interference type includes: If the interference level is less than the first preset threshold, the target imaging mode is determined to be the first imaging mode; If the interference level is greater than or equal to a first preset threshold and less than a second preset threshold, the target imaging mode is determined to be a second imaging mode or a third imaging mode according to the type of interference; the first preset threshold is less than the second preset threshold. When the interference level is greater than or equal to the second preset threshold, or when the difference between the probability of the interference type being the first type and the probability of the interference type being the second type is within a preset range, the target imaging mode is determined to be the second imaging mode and the third imaging mode are executed sequentially; the distribution pattern of the condensate in the second type is more discrete than the distribution pattern of the condensate in the first type.

3. The method according to claim 2, characterized in that, The step of determining whether the target imaging mode is a second imaging mode or a third imaging mode based on the type of interference includes: If the interference type is the first type, the target imaging mode is determined to be the second imaging mode; If the interference type is the second type, the target imaging mode is determined to be the third imaging mode.

4. The method according to claim 2, characterized in that, The target feature vector includes a shape feature vector; the step of acquiring a second image of the object to be identified in the target imaging mode, and determining the object category based on the target feature vector corresponding to the second image, includes: When the target image mode is the second imaging mode, the visible light is turned off, and the invisible light of the second preset wavelength is projected onto the surface of the object to be identified in the form of a preset pattern, and the second image of the object to be identified is acquired. Based on the second image, a 3D point cloud of the object to be identified is constructed; Shape feature vectors are extracted from the three-dimensional point cloud, and the item category of the object to be identified is determined based on the shape feature vectors.

5. The method according to claim 2, characterized in that, The target feature vector includes a texture feature vector; the step of acquiring a second image of the object to be identified in the target imaging mode, and determining the object category of the object to be identified based on the target feature vector corresponding to the second image, includes: When the target imaging mode is the third imaging mode, the polarizer is set to the first direction to acquire the first candidate image of the object to be identified; The polarizer is set to the second direction to acquire a second candidate image of the object to be identified; the first direction and the second direction are orthogonal to each other. Calculate the difference image between the first candidate image and the second candidate image, and determine the difference image as the second image; Extract texture feature vectors from the second image, and determine the item category of the item to be identified based on the texture feature vectors.

6. The method according to claim 2, characterized in that, The target feature vector includes a first feature vector and a second feature vector; the step of acquiring a second image of the object to be identified in the target imaging mode, and determining the object category based on the target feature vector corresponding to the second image, includes: In the second imaging mode, a third candidate image of the object to be identified is acquired and a corresponding first feature vector is determined based on the third candidate image; In the third imaging mode, a fourth candidate image of the object to be identified is acquired and a corresponding second feature vector is determined based on the fourth candidate image; Based on the first feature vector and the second feature vector, the item category of the item to be identified is determined.

7. The method according to claim 6, characterized in that, Determining the item category of the item to be identified based on the first feature vector and the second feature vector includes: Based on the confidence scores and historical recognition results of the first feature vector and the second feature vector in the current environment, respectively, corresponding fusion weights are assigned to the first feature vector and the second feature vector; The first feature vector and the second feature vector are weighted and fused according to the fusion weight to obtain a fused feature vector; The item category of the item to be identified is determined based on the fused feature vector.

8. The method according to claim 1, characterized in that, Determining the degree and type of interference caused by condensation to the first image includes: The first image is input into the evaluation network, which outputs the degree of interference and the type of interference. The evaluation network includes a first evaluation sub-network and a second evaluation sub-network. The first evaluation sub-network is used to output the degree of interference, and the second evaluation sub-network is used to output the type of interference. The training samples of the first evaluation sub-network include multiple images acquired under various condensation conditions under visible light illumination at the first preset wavelength, and the degree of interference is labeled on the multiple images. The training samples of the second evaluation sub-network include multiple images acquired under various condensation conditions under visible light illumination at the first preset wavelength, and the type of interference is labeled on the multiple images.

9. The method according to claim 1, characterized in that, The method further includes: Determine the confidence level of this identification process; If the confidence level is greater than or equal to the preset success threshold, the identification process is determined to be complete. If the confidence level is less than the preset success threshold, the first image of the item to be identified is re-acquired or a prompt message is output. After receiving the operation in response to the prompt message, the first image of the item to be identified is re-acquired.

10. An item identification device, characterized in that, The device includes: The image evaluation module is used to acquire a first image of the object to be identified and to determine the degree and type of interference caused by condensation in the first image. An imaging mode determination module is used to determine a target imaging mode from a first imaging mode, a second imaging mode, and a third imaging mode based on the interference level and the interference type; the first imaging mode is a mode for acquiring images under visible light illumination, the second imaging mode is a mode for acquiring images under invisible light illumination, and the third imaging mode is a mode for acquiring images with the polarizer set at at least two different angles. The category determination module is used to acquire a second image of the object to be identified in the target imaging mode, and determine the object category of the object to be identified based on the target feature vector corresponding to the second image.

11. An electronic device, characterized in that, The electronic device includes a memory and one or more programs, wherein one or more programs are stored in the memory and configured to be executed by one or more processors according to any one of claims 1 to 9.

12. A readable storage medium, characterized in that, When the instructions in the readable storage medium are executed by a processor of an electronic device, the processor is enabled to perform the method as described in any one of claims 1 to 9.

Images