Vending machine multi-commodity selling price calculation method and system based on AI visual identification

Abstract

The invention relates to the field of visual identification, in particular to a vending machine multi-commodity selling price calculation method and system based on AI visual identification. The method comprises the following steps: collecting a three-dimensional image sequence in the vending machine, and carrying out panoramic image edge splicing to obtain a corrected panoramic spliced image; performing multi-scale image noise filtering on the corrected panoramic spliced image to obtain a filtering optimized panoramic image; performing one-by-one commodity visual identification on the filtered and optimized panoramic image, performing spatial commodity distribution analysis, and constructing a three-dimensional commodity distribution model; identifying a door opening signal of the vending machine, collecting a newest internal detection image, and performing differential comparative analysis on the three-dimensional commodity distribution model to obtain a commodity change list; and performing multi-price parallel calculation on the commodity change list, and outputting a price list. Through efficient and accurate price calculation of the vending machine, detailed price information is output, and the purchase experience of the user is improved.

Description

Technical Field

[0001] This invention relates to the field of visual recognition, and in particular to a method and system for calculating the selling price of multiple items in a vending machine based on AI visual recognition. Background Technology

[0002] In practical applications involving the sale of multiple products, traditional vending machines rely on fixed tracks, barcode recognition, or single-item detection for product identification and pricing. These methods have significant limitations when faced with complex shopping scenarios, especially when multiple items are purchased at once, product placement is not fixed, or labels are damaged. Traditional systems are prone to identification errors, inaccurate pricing, and transaction failures, severely impacting user experience and equipment operational efficiency. Furthermore, with the continuous enrichment of product varieties and the increasing personalization needs of users, vending machines not only need to support diverse product identification but also need to implement real-time pricing, price updates, and promotional discounts for different products. Traditional static pricing methods are no longer sufficient to meet current market demands. Simultaneously, in unattended operating environments, accurately judging the product selection process and preventing risks such as theft and underpayment are also urgent technical challenges that need to be addressed. Summary of the Invention

[0003] To address the aforementioned technical problems, this invention proposes a method and system for calculating the selling prices of multiple items in a vending machine based on AI visual recognition, thereby resolving at least one of the aforementioned technical issues.

[0004] To achieve the above objectives, this invention provides a method for calculating the selling price of multiple items in a vending machine based on AI visual recognition, comprising the following steps: Step S1: Collect a three-dimensional image sequence inside the vending machine, perform panoramic image edge stitching, and obtain a corrected panoramic stitched image; Step S2: Perform multi-scale image noise filtering on the corrected panoramic stitched image to obtain a filtered and optimized panoramic image; Step S3: Perform visual recognition of each product in the filtered and optimized panoramic image, analyze the spatial distribution of products, and construct a three-dimensional product distribution model; Step S4: Identify the vending machine's door opening signal, collect the latest internal detection images, and perform differential comparison analysis on the three-dimensional product distribution model to obtain a list of product changes; Step S5: Perform parallel calculations on multiple prices for the product change list and output the price list.

[0005] This specification provides a vending machine multi-item sales price calculation system based on AI visual recognition, used to execute the vending machine multi-item sales price calculation method based on AI visual recognition as described above, including: The panoramic stitching module is used to acquire a sequence of three-dimensional images inside the vending machine, stitch the panoramic images at the edges, and obtain a corrected panoramic stitched image. The image optimization module is used to perform multi-scale image noise filtering on the corrected panoramic stitched image to obtain a filtered and optimized panoramic image. The product distribution module is used to perform visual recognition of each product in the filtered and optimized panoramic image, analyze the spatial distribution of products, and build a three-dimensional product distribution model. The differential comparison module is used to identify the vending machine's door opening signal, collect the latest internal detection images, and perform differential comparison analysis on the three-dimensional product distribution model to obtain a list of product changes. The parallel computing module is used to perform parallel calculations on multiple prices for the product change list and output a price list.

[0006] The beneficial effects of this invention are as follows: By acquiring a sequence of three-dimensional images of the vending machine's interior from multiple angles and stitching them together to form a complete corrected panoramic image, the occlusion, blind spots, and distortion problems caused by traditional single-view images are effectively overcome. The construction of the panoramic image allows product images to be displayed under a unified field of view, greatly improving the accuracy and consistency of subsequent visual recognition. Through multi-scale image noise filtering (such as wavelet denoising, bilateral filtering, or guided filtering), high-frequency noise can be effectively removed while preserving edge information, significantly improving image quality. The advantage of multi-scale processing is that it can simultaneously consider both details and global features, removing background interference (such as stickers, dust, and light reflections) while retaining product boundary lines, providing a clear input image for subsequent product segmentation and recognition. By using AI visual recognition algorithms to locate products one by one (combining target detection and instance segmentation techniques), the system can accurately identify the type, quantity, and relative position of products in the aisle. Combined with the three-dimensional images acquired from multiple angles, the three-dimensional coordinates of the products can be further reconstructed, constructing a complete three-dimensional product distribution model. By detecting door opening actions (which can be identified by a door magnetic sensor or a built-in control system) as the trigger point for image updates, the real-time nature and event-driven nature of data acquisition are ensured. The new image is compared differentially with the previously constructed 3D product distribution model to quickly locate changes in the quantity or location of products, achieving precise judgment of "what's missing." The comparative analysis employs a pixel-by-pixel / point-by-point differential mechanism, combined with position matching and shape change judgment, effectively avoiding false alarms caused by lighting or occlusion. This is a crucial step in converting visual recognition results into specific transaction prices. The product change list already contains the product's unique identifier (SKU code), changed quantity, and aisle location information. The system calls the local product pricing database or a network price interface (such as a cloud-based price strategy management platform) to perform multi-dimensional parallel pricing calculations, incorporating factors such as promotional strategies, time-based pricing, and user levels. Through a parallel algorithm framework (such as GPU parallel or multi-threaded computing), even when multiple products are identified simultaneously (such as retrieving three beverages at once), an accurate total price can be output within 300ms. Attached Figure Description

[0007] Figure 1 This is a flowchart illustrating the steps of a method for calculating the selling price of multiple items in a vending machine based on AI visual recognition, according to the present invention. Figure 2 This is a detailed flowchart illustrating the implementation steps of step S1. Figure 3 This is a detailed flowchart illustrating the implementation steps of step S2; Figure 4 This is a flowchart illustrating the detailed implementation steps of step S3. Detailed Implementation

[0008] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0009] This application provides a method and system for calculating the prices of multiple items in a vending machine based on AI visual recognition. The executing entities of the method and system for calculating the prices of multiple items in a vending machine based on AI visual recognition include, but are not limited to, mechanical equipment, data processing platforms, cloud server nodes, and network upload devices that can be considered as general computing nodes in this application. The data processing platform includes, but is not limited to, at least one of an audio-visual management system, an information management system, and a cloud-based data management system.

[0010] Please see Figures 1 to 4 This invention provides a method for calculating the selling price of multiple items in a vending machine based on AI visual recognition, including the following steps: Step S1: Collect a three-dimensional image sequence inside the vending machine, perform panoramic image edge stitching, and obtain a corrected panoramic stitched image; Step S2: Perform multi-scale image noise filtering on the corrected panoramic stitched image to obtain a filtered and optimized panoramic image; Step S3: Perform visual recognition of each product in the filtered and optimized panoramic image, analyze the spatial distribution of products, and construct a three-dimensional product distribution model; Step S4: Identify the vending machine's door opening signal, collect the latest internal detection images, and perform differential comparison analysis on the three-dimensional product distribution model to obtain a list of product changes; Step S5: Perform parallel calculations on multiple prices for the product change list and output the price list.

[0011] In the embodiments of the present invention, see Figure 1 This is a flowchart illustrating the steps of a method for calculating the prices of multiple items in a vending machine based on AI visual recognition, according to the present invention. In this example, the steps of the method for calculating the prices of multiple items in a vending machine based on AI visual recognition include: Step S1: Collect a three-dimensional image sequence inside the vending machine, perform panoramic image edge stitching, and obtain a corrected panoramic stitched image; In this embodiment, a high-resolution camera array of 4-6 cameras is deployed inside the vending machine, using a resolution of 1920×1080 pixels and a frame rate of 30fps to ensure time synchronization of image acquisition. The cameras are arranged in a ring at 45-degree intervals, covering a 360-degree field of view inside the vending machine. Each camera is equipped with a 28mm equivalent focal length lens, an aperture of F2.8, an ISO sensitivity controlled within the range of 400-800, and a shutter speed of 1 / 60 second to ensure image clarity and consistency. Simultaneously, a depth sensor array is activated to acquire depth information with a distance accuracy of ±2mm, with a depth measurement range of 0.3-2.0 meters, matching the dimensions of the vending machine's interior space. Geometric distortion correction is performed on the acquired multi-angle image sequences, using a pre-calibrated intrinsic parameter matrix to eliminate radial and tangential distortion. The SIFT feature detection algorithm is used to extract key feature points from each image, with a feature point detection threshold set to 0.04 and an edge threshold set to 10 to ensure stable corner feature extraction. Feature point matching was performed on adjacent viewpoint images using a FLANN matcher for fast nearest neighbor search, with a distance ratio threshold of 0.7. False matches were removed using the RANSAC algorithm, with an interior point threshold of 3 pixels and a confidence level of 0.99. The relative pose relationships between cameras were calculated based on the matched feature point pairs, and rotation matrices and translation vectors were obtained using essential matrix decomposition. A dense 3D point cloud was reconstructed using triangulation, with a point cloud density controlled at 100-200 points per square centimeter. A spherical projection model was used to project the 3D scene onto a spherical coordinate system, with a sphere radius of 1 meter and a latitude / longitude resolution of 0.1 degrees. Pixel resampling was performed using bicubic interpolation to generate a 4096×2048 pixel spherical panoramic image. The stitching boundaries between adjacent images were detected, and a gradient change detection algorithm was used to identify the stitching seam location, with a gradient threshold of 30. A 15-25 pixel wide stitching transition band was established, and seamless stitching was achieved using multi-band fusion technology. The Laplacian pyramid decomposition method was used, with 5 decomposition layers. Distance-weighted fusion was applied to low-frequency components, using a Gaussian distribution as the weighting function and setting the standard deviation to 1 / 3 of the transition band width. Selective fusion was performed on high-frequency components to preserve details in areas with higher signal-to-noise ratios. A geometrically corrected panoramic stitched image was generated, with stitching errors controlled within 2 pixels.

[0012] Step S2: Perform multi-scale image noise filtering on the corrected panoramic stitched image to obtain a filtered and optimized panoramic image; In this embodiment, noise type detection and analysis are performed on the output corrected panoramic stitched image. Wavelet transform is used to decompose the image into four scale levels, each scale containing detail coefficients in three directions: horizontal, vertical, and diagonal. By statistically analyzing the variance and kurtosis parameters of the detail coefficients at each scale, different types of noise, such as Gaussian noise and stitching noise, are identified. The detection threshold for Gaussian noise is set to a variance greater than 100 and a kurtosis less than 3.5. Salt-and-pepper noise detection is based on pixel value abrupt change detection, and the abrupt change threshold is set to a difference between adjacent pixels greater than 80. Stitching noise is mainly distributed within a 20-pixel range near the stitching boundary and is identified through local variance analysis, with a variance threshold set to 150. Corresponding filtering algorithms are used to process different types of noise. For Gaussian noise, a nonlocal mean filtering algorithm is used, with the search window size set to 21×21 pixels, the similarity window size set to 7×7 pixels, and the filtering intensity parameter h set to 0.1 times the noise standard deviation. Median filtering is used for salt-and-pepper noise, with the filter kernel size adaptively adjusted according to noise density: a 3×3 kernel for noise density less than 5%, a 5×5 kernel for density between 5% and 15%, and a 7×7 kernel for density greater than 15%. Anisotropic diffusion filtering is used for splicing noise, with a diffusion coefficient K set to 30, an iteration count of 10, and a time step of 0.125. An edge protection strategy is employed during noise filtering to prevent the loss of important product edge information. The Canny edge detection algorithm is used to pre-extract image edges, with a low threshold of 50, a high threshold of 150, and a Gaussian kernel size of 5×5. The filtering intensity is reduced within 5 pixels of the edge pixels, with the filtering parameters adjusted to 0.6 times the original value. Edge protection is performed using a bilateral filter, with the spatial Gaussian kernel standard deviation set to 5, the grayscale Gaussian kernel standard deviation set to 50, and the filter kernel size set to 9×9 pixels. The quality of the filtered image is evaluated using Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM) as evaluation metrics. The PSNR target value is set above 30dB, and the SSIM target value is set above 0.85. If the evaluation result does not meet the target value, the filtering parameters are adjusted for iterative optimization. Simultaneously, the BRISQUE no-reference image quality assessment method is used for comprehensive quality evaluation, with a score range of 0-100. A target value below 25 indicates good quality. For local areas with substandard quality, block-based adaptive filtering is used for reprocessing, with a block size of 32×32 pixels and an overlap of 50%. The output filter optimizes the panoramic image, controlling the noise level to below 20% of the original image.

[0013] Step S3: Perform visual recognition of each product in the filtered and optimized panoramic image, analyze the spatial distribution of products, and construct a three-dimensional product distribution model; In this embodiment, a deep learning object detection algorithm is used for product recognition based on the filtered and optimized panoramic image. YOLOv5 or Faster R-CNN models are deployed, and the input image size is standardized to 640×640 pixels. Model training uses a dataset containing 500 common product categories, with 50,000 training samples and a validation set accounting for 20%. The detection confidence threshold is set to 0.5, and the IoU threshold for Non-Maximum Suppression (NMS) is set to 0.4. Accurate segmentation of the detected product bounding boxes is performed using Mask R-CNN or DeepLab v3+ semantic segmentation algorithms, achieving pixel-level accuracy. A corresponding binary mask is generated for each product target, and the mask accuracy is evaluated using the IoU metric, with a target value set above 0.8. Multi-dimensional feature descriptors are extracted from each segmented product region, including three main categories: geometric features, color features, and texture features. Geometric features include seven parameters such as the product's aspect ratio, area, perimeter, and roundness, with area calculation accuracy reaching the pixel level. Color features are derived using the HSV color space, extracting the mean and standard deviation of hue (H), saturation (S), and lightness (V), totaling six parameters. Texture features are derived using the Gray-Level Co-occurrence Matrix (GLCM) method, calculating four texture parameters: contrast, correlation, energy, and entropy. The calculation directions are set to 0°, 45°, 90°, and 135°, with a distance of 1 pixel. The extracted 17-dimensional feature vectors are normalized using Z-score standardization to ensure consistent numerical ranges for each dimension. The geometric relationship between panoramic images and 3D space is utilized to convert 2D image coordinates into 3D physical coordinates. A transformation model from spherical to Cartesian coordinates is established, with the sphere radius determined based on depth sensor data, achieving millimeter-level accuracy. The centroid coordinates of each product are located in 3D space with coordinate accuracy controlled within ±3mm. Considering the spatial volume occupied by products, a standard size database is established based on product categories, including length, width, and height parameters, covering 95% of common products. The precise position and orientation of goods in 3D space are calculated using inverse perspective projection transformation, with the orientation angle accuracy controlled within ±2 degrees. Based on the 3D spatial coordinates and feature information of the goods, a hierarchical goods distribution model is constructed. A three-level hierarchical structure is established: shelf layer, aisle layer, and goods layer, each containing corresponding spatial indexes and attribute information. The shelf layer records the overall dimensions of the vending machine, with length × width × height set at 1200 × 800 × 1800 mm. The aisle layer is divided into an 8 × 6 grid structure, with each grid cell measuring 150 × 133 mm, recording the quantity and category distribution of goods within the grid. The goods layer records the unique ID, category label, 3D coordinates, feature vector, price information, and other attributes for each goods. A spatial adjacency graph between goods is established, with an adjacent distance threshold set at 50 mm for subsequent change detection and occlusion analysis.The completed 3D commodity distribution model contains global information such as the total number of commodities, spatial distribution density, and category statistics, providing benchmark data for subsequent differential comparative analysis.

[0014] Step S4: Identify the vending machine's door opening signal, collect the latest internal detection images, and perform differential comparison analysis on the three-dimensional product distribution model to obtain a list of product changes; In this embodiment, A dual detection mechanism, employing both a magnetic proximity sensor and a mechanical limit switch, monitors the opening status of the vending machine door. The magnetic sensor's detection distance is set to 15mm, with a response time of less than 10ms. It immediately triggers the image acquisition system when the door moves away from its closed position. The mechanical limit switch serves as a backup detection mechanism, with an actuation force set to 2N to ensure reliable detection. After the door opening signal is triggered, a high-speed image acquisition mode is activated, increasing the frame rate to 60fps and shortening the exposure time to 1 / 120th of a second to capture dynamic changes during the user's product retrieval process. The acquisition duration is set to the complete user operation cycle, from door opening to closing, typically lasting 30-120 seconds. Simultaneously, an infrared supplementary lighting system is activated, with a supplementary light power set to 50W and a color temperature set to 5500K to ensure consistent illumination during image acquisition. The latest acquired image sequence undergoes the same preprocessing steps as S1 and S2, including distortion correction, panoramic stitching, and noise filtering. The last frame image after the door closes is selected as the comparison benchmark to ensure image stability and clarity. The same product detection algorithm is used to identify products in the new image, and the detection parameters remain consistent with step S3 to ensure comparability of results. Detected products are renumbered and their features extracted to create a new product distribution list. Processing time is controlled within 3-5 seconds to meet the real-time requirements of user payments. GPU acceleration and CUDA parallel processing technology are used to increase the detection speed by 5-8 times. A precise comparative analysis is performed between the baseline 3D product distribution model and the new detection results. A product correspondence establishment method based on feature matching is used, with a matching threshold set to a feature vector Euclidean distance of less than 0.3. Position change analysis is performed on successfully matched products, with a position change threshold set at 30mm; exceeding this threshold indicates product movement. Baseline products that cannot be matched are marked as "suspected to have been taken," and newly added products are marked as "suspected to have been placed." Occlusion analysis is performed using spatial adjacency to determine if product disappearance is caused by occlusion from other products. Occlusion judgment is based on a line-of-sight occlusion algorithm, calculating the intersection of the line of sight and the product bounding box. Connectivity analysis is used to handle changes in product cluster areas, with a connected component size threshold set at 200 pixels to ensure that minor positional adjustments are not misinterpreted as product removal. Based on differential comparison analysis results, a structured product change list is generated, including product ID, product name, change type (removal / placement / movement), change quantity, and confidence level. Confidence level calculation comprehensively considers factors such as feature matching degree, positional change magnitude, and occlusion analysis results, using a weighted average method with weights set to 0.4, 0.3, and 0.3 respectively. Changes with confidence levels below 0.7 undergo secondary verification by tracking product movement trajectories using time-series image analysis to verify the authenticity of the changes. Cross-validation is performed using weight sensor data, with a weight change threshold set at 80% of a single product's weight, a weight data sampling frequency of 100Hz, and a 5-point average filter.The generated list of product changes is sorted from highest to lowest confidence level. Changes with a confidence level greater than 0.85 are directly included in the list, changes with a confidence level between 0.7 and 0.85 are marked as "requires confirmation," and changes with a confidence level less than 0.7 are marked as "suspected false positive." The output includes a list of changes for which product information has been confirmed to be retrieved, with an accuracy target set at 95% or higher.

[0015] Step S5: Perform parallel calculations on multiple prices for the product change list and output the price list.

[0016] In this embodiment, the output list of product changes undergoes product code parsing to extract detailed attributes such as SKU identifier, product name, brand information, and specifications for each changed product. A local product price database is established, containing price information for 50,000 common products. The database is stored using MySQL and uses a B+ tree index structure based on SKU, with query response time controlled within 10ms. The price database is synchronized with the headquarters pricing system in real time, with a synchronization frequency set to once per hour to ensure the accuracy and timeliness of price information. For products not found in the database, an online price query mechanism is initiated, querying the headquarters system via an API interface. The API response time is set to within 500ms; if a timeout occurs, a default price is used or manual confirmation is prompted. A multi-threaded parallel computing architecture is used to process product price calculation tasks, with a thread pool size of 8 threads, each thread independently processing the price calculation for one product. For batch product calculations, a MapReduce computing model is used. The Map phase queries the unit price of each product in parallel, and the Reduce phase aggregates and calculates the total price. The price calculation for a single product includes multiple components such as the base price, promotional discounts, membership benefits, and taxes. Promotional discount queries involve rules for time-limited promotions, quantity discounts, and bundled offers. The rule engine is implemented using Drools, with a rule response time controlled within 50ms. Member discounts are calculated based on user level and points, with the discount rate ranged from 0-20%, and the calculation precision maintained to two decimal places. Multiple verification checks are performed on the calculated price results to ensure accuracy and reasonableness. A price reasonableness verification mechanism is established, with the price range for a single product set at 0.5-500 yuan; exceeding this range triggers an anomaly alert. The total price is compared with historical transaction data; if the total price exceeds three times the standard deviation of the historical average, it is marked as an abnormal transaction requiring manual confirmation. Price verification is performed using weight sensor data; the theoretical total weight is estimated based on the average weight of the product and compared with the actual weight change; a weight deviation exceeding 20% ​​triggers a price recalculation. A price calculation log recording mechanism is established to record the detailed process and intermediate results of each calculation, facilitating anomaly tracking and auditing. The verified price calculation results are formatted into a standard price list, including complete information such as product name, product image, unit price, quantity, subtotal, discount amount, and total amount payable. The price list is stored in a structured JSON format and supports multiple front-end display methods. The price details are displayed in a table format on the user interface, with 5 columns, each row height set to 60 pixels, and a font size of 14px to ensure clear readability. The total price is displayed in prominent red font with a font size of 24px, and a function to convert the amount to capital letters is added. A secondary confirmation mechanism for the price details is provided; users can click to view detailed information and price breakdown for each item. Multiple payment methods are supported, including WeChat Pay, Alipay, and UnionPay cards, with payment interface response time controlled within 2 seconds.Transaction records containing timestamps, product lists, and price details are generated and stored in a local database and cloud system, ensuring the integrity and traceability of transaction data. The output price list has an accuracy rate of over 99.5%, and the calculation response time is controlled within 3 seconds, meeting users' needs for rapid payment.

[0017] In this embodiment, see Figure 2 The diagram below illustrates the detailed implementation steps of step S1. In this embodiment, the detailed implementation steps of step S1 include: Multi-angle stereoscopic imaging of the internal three-dimensional space of the vending machine is performed to acquire a sequence of three-dimensional images. Image feature point recognition is performed on a 3D image sequence, and image feature points from multiple angles are marked; Multi-angle visual registration is performed based on image feature points from multiple angles to obtain multi-angle visual registration parameters; Image overlapping region detection and extraction are performed based on multi-angle visual registration parameters; Calculate the geometric transformation parameters of the overlapping region to obtain the rotation, translation, and scaling relationships between the images; The three-dimensional image sequence is edge-stitched according to the rotation, translation and scaling relationships to obtain a corrected panoramic stitched image.

[0018] In this embodiment, multiple RGB-D cameras or stereo cameras, such as Intel RealSense D435i or Azure Kinect, are deployed on the top, left and right sides, and front of the vending machine. These cameras can capture RGB images and depth information, covering all product display layers inside the vending machine through imaging from multiple perspectives. In the experimental design, the number of cameras is generally 4 to 6, fixed at a distance of about 30cm in front of the products, with an angle of 30 to 45 degrees between each camera to ensure sufficient overlap between image perspectives. The cameras use a synchronous triggering mechanism to capture 30 frames per second, with an image resolution of at least 1280×720 to preserve clear product details. The depth map and RGB image are paired with timestamps and combined into a point cloud sequence, which is stored as a complete 3D image sequence data package, providing a data foundation for subsequent image feature analysis and geometric modeling. The real-time performance and accuracy of this acquisition process directly determine the effectiveness of the subsequent recognition system in acquiring information on product type, quantity, and location, and are a fundamental step in building a high-quality product recognition model. The ORB (Oriented FAST and Rotated BRIEF) algorithm is employed, a highly efficient and computationally-friendly image feature extraction method suitable for real-time systems. In each frame of the RGB image, corner points are first detected using the FAST algorithm, and then oriented and scaled using the BRIEF descriptor, forming a set of highly recognizable feature points. In the experimental setup, approximately 800-1200 feature points are extracted from each image, distributed across product outlines, label boundaries, barcode lines, and high-frequency areas of patterns, ensuring high stability and reproducibility of the features. For image sequences acquired from different angles, viewpoint labels and image frame indices are added to each feature point for cross-view matching. After feature point extraction, non-maximum suppression and feature filtering are performed to remove duplicate, blurred, or noisy points, forming a feature point database usable for image registration. This process directly affects the spatial registration accuracy between images and is a prerequisite for image fusion and multi-view modeling.

[0019] Spatial registration between images is achieved by calculating relative positional relationships through matching feature points across different viewpoints. The image registration process employs a RANSAC (Random Sample Consensus)-based feature point matching strategy, combined with brute-force matching or a fast search algorithm based on FLANN, to identify the most likely matching feature pairs between different images and filter out abnormal matches. In each image matching group, the extracted matching feature pairs are used to estimate the homography matrix or rigid transformation matrix (including rotation and translation) to describe the spatial transformation relationship between images. To achieve high-precision multi-angle registration, a bundle adjustment algorithm is used to globally optimize the entire image sequence, ensuring that all viewpoint images simultaneously satisfy spatial consistency constraints. In experiments, approximately 300–500 high-quality matching points are retained for each image pair, and the optimization iteration terminates when the error converges to less than 0.01 pixels. The system outputs a set of multi-angle visual registration parameters, including the rotation vector, translation vector, and scaling factor for each image, providing a precise spatial alignment basis for subsequent image fusion and overlapping region detection. By mapping all viewpoint images to a unified reference viewpoint using the obtained geometric transformation parameters, overlapping regions between images are identified. Overlap detection primarily relies on image boundary mapping analysis and feature point distribution density judgment. The system projects all images using the transformation matrix and calculates the coverage area of ​​each image boundary in the reference coordinate system, identifying image pairs with an overlap exceeding a certain threshold (typically set to 25% or more of the image area). Subsequently, by comparing the number and distribution similarity of feature points in the overlapping regions, and using metrics such as pixel histogram intersection and structural similarity (SSIM), the system filters out image regions that actually overlap. In experimental tests, the average number of feature point matches in effective overlapping regions was 150–300 pairs, with a structural similarity greater than 0.8. Accurate extraction of these regions is crucial for subsequent image fusion and seam processing. To improve image fusion quality, the system also records information such as image overlap, boundary deviation, and color difference for each overlapping region, for use in subsequent image stitching and illumination correction.

[0020] Further calculations of the specific geometric transformation relationships in these regions are performed to achieve precise alignment and fusion between images. The system employs an affine transformation modeling method, taking the coordinates of matched feature point pairs in the overlapping regions as input. Least squares fitting is used to derive the rotation angle, translation distance, and scaling factor between images. This process first constructs linear transformation equations between 2D point sets, then uses the singular value decomposition (SVD) algorithm to solve the coefficient matrix, separating the rotation matrix and translation vector. In the experimental parameter settings, the system requires each overlapping region to contain at least 200 valid point pairs, and the fitting error must be controlled within 1.5 pixels. To improve robustness to nonlinear distortion, a robust loss function (such as Tukey loss) is introduced to penalize outliers, improving model stability. The output is a set of geometric transformation parameters containing rotation angles (e.g., 3.4°), translation vectors (e.g., 15px in the x-direction, -8px in the y-direction), and scaling factors (e.g., 1.02). These parameters are used to geometrically correct each image during image stitching, ensuring seamless alignment of all images in the overlapping regions.

[0021] Based on the obtained rotation, translation, and scaling parameters, all images undergo geometric correction and are uniformly mapped onto a common coordinate plane. The system smooths image edge pixels using image interpolation (such as bilinear interpolation) to avoid jagged edges and holes caused by geometric transformations. For overlapping areas, a multi-scale Laplacian fusion algorithm or pyramid fusion strategy is employed to achieve consistent color, brightness, and texture processing, preventing obvious gaps or color jumps at image boundaries. In practice, dynamic weighted fusion is used in the image stitching area, with lower weights closer to the boundary, thereby improving visual coherence. The stitched image resolution can reach over 3000×2000 pixels, retaining all product display information and forming a clear and unified panoramic image for AI recognition model input. This stitched image serves as the core visual input for subsequent real-time product detection, recognition, and price calculation, playing a decisive role in achieving full coverage of product information within the vending machine and improving recognition accuracy.

[0022] In this embodiment, the specific steps for performing edge stitching processing on the three-dimensional image sequence based on the rotation, translation, and scaling relationships to obtain the corrected panoramic stitched image are as follows: Based on the rotation, translation, and scaling relationships, multi-angle image edge detection is performed on the three-dimensional image sequence to identify the angular boundaries between different images; Calculate the pixel gradient at the angular boundary to determine the image abrupt stitching seam; Morphological refinement of the stitching seams in the image abrupt changes yields stitching edge lines with a width of one pixel. Pixel-level point-by-point calculations are performed on the spliced ​​edge lines to obtain pixel-level coordinate indices; Multi-band image stitching is performed based on pixel-level coordinate indexing to obtain a three-dimensional stitched image; Perform splicing geometric distortion identification on 3D stitched images and mark the locations of geometric distortions; Local geometric correction is performed on the locations of geometric distortions, and bilinear interpolation resampling is used to construct a corrected panoramic stitched image.

[0023] In this embodiment, after obtaining the rotation, translation, and scaling relationships, the system can unify the 3D image sequence into a common coordinate system, facilitating the analysis of stitching boundaries between images from different viewpoints. The core of this step is to detect edge regions in multi-angle images and identify angular boundaries between images, i.e., edge regions where viewpoints abruptly change. The system performs affine transformations on all images according to known geometric parameters, and then uses the Canny edge detection algorithm to extract edges from each image. To improve the accuracy of edge recognition, the images are pre-blurred using Gaussian blur (sigma=1.2, kernel=5×5) to reduce noise interference. In the processed edge map, by comparing pixel intensity changes with the positions of overlapping areas, the system can calculate the edge clustering zone at the image boundaries, i.e., the region with the most significant viewpoint changes. Furthermore, by grouping edge points using a clustering algorithm (such as DBSCAN), it can distinguish edge points belonging to the same viewpoint image from angular boundary lines belonging to the boundaries of different images. The output is an angular boundary image mask between multiple images, which provides the basic recognition structure for subsequent stitching seam detection and geometric distortion processing.

[0024] Further analysis of the pixel gradient distribution in these boundary regions is used to identify potential image stitching abrupt changes. Abrupt changes refer to regions of grayscale or structural discontinuity that occur at the boundaries after image stitching, typically manifested as significant brightness jumps or texture interruptions. In this step, the system first extracts the grayscale image of the original image from the angular boundary mask region, and then uses the Sobel operator in its boundary region to calculate the pixel gradients in the horizontal and vertical directions (Gx and Gy, respectively), thereby obtaining a gradient magnitude map (using the formula √(Gx²+Gy²)). To better distinguish between natural textures and abrupt changes, the system sets a gradient intensity threshold (e.g., 25%~30% of the maximum image gradient value). Boundary regions exceeding this threshold are identified as abrupt changes. In the experiment, a 3×3 window is used for local sliding calculations, and the mean and variance of the local gradient are calculated for each region to determine boundary continuity. If the gradient difference is greater than a set standard deviation (e.g., σ>20), the region is marked as a potential stitching seam. Through this process, the system can accurately locate discontinuous gaps in image stitching caused by problems such as viewing angle errors, uneven brightness, or camera occlusion, providing target boundaries for subsequent image edge refinement.

[0025] The stitching seam regions obtained in the previous stage are converted into binary images (seam regions are 1, other regions are 0). Then, image skeleton extraction methods provided by OpenCV or image processing libraries, such as the Zhang-Suen algorithm or a Hilditch-based thinning algorithm, are used to perform morphological thinning on these regions. During the thinning process, the system iterates through local 3×3 pixel regions multiple times, gradually removing redundant pixels, preserving the structural backbone, and avoiding structural breaks. In experimental parameters, the maximum number of iterations is typically controlled to 20 rounds, or until the structure no longer changes. The result is a single-pixel-wide stitching edge line that clearly represents the boundary seam path between different images. This edge line not only improves the efficiency of subsequent pixel-level calculations but also reduces error propagation caused by boundary blurring, serving as a key input for high-precision image stitching and distortion correction.

[0026] The algorithm iterates through each non-zero pixel in the stitched edge image and records its coordinates (x, y) in the original image. Simultaneously, it combines the original viewpoint image index of the pixel and its position in 3D space (if depth data is available) to create a coordinate index table containing the original pixel position, the projected target position, and the viewpoint number. In the experimental configuration, a single stitched edge line typically contains hundreds to thousands of pixels. To avoid data redundancy, path compression is performed on the point set, and redundant points with a distance less than 2 pixels are removed. This pixel-level coordinate index structure is a sparse matrix form, supporting fast indexing, parallel processing, and region interpolation operations. Gaussian and Laplacian pyramids (typically 5-6 layers) are constructed for the images involved in the stitching, decomposing the image information into representations of different spatial frequencies. Then, based on the pixel-level coordinate index, the corresponding frequency band information in the different viewpoint images is mapped to the corresponding positions in the stitched image. Finally, all images are weighted and fused in each frequency band, with weights allocated according to image quality, shooting angle, and overlap. The complete 3D stitched image is obtained through pyramid reconstruction. In the experimental setup, 3×3 or 5×5 convolutional kernels were used to construct the pyramid, with each frequency band retaining at least 70% feature consistency in the stitched region. The result is a 3D stitched image that integrates texture, color, and structural information from multiple viewpoints, providing high-quality input for the AI ​​model and greatly improving the robustness and accuracy of product recognition.

[0027] Distortion analysis is performed on the entire 3D stitched image to identify geometric distortion regions caused by image mismatch, parallax errors, or projection inconsistencies. Distortion identification is divided into two levels: structural distortion detection, which mainly uses image gradient flow consistency analysis to determine whether the continuity of the image structure is disrupted; and texture distortion detection, which uses local autocorrelation analysis to determine whether the texture is stretched or compressed. In the specific implementation, Harris corner density and gradient direction histogram analysis are used to generate gradient vector fields in the edge seam regions of the image. Then, abrupt changes in gradient direction are detected, and local direction consistency indices (such as the eigenvalue ratio of the structure tensor) are calculated. If the index is lower than a set threshold (e.g., consistency metric less than 0.6), the region is marked as a geometric distortion location. In the experiment, an average of 5-10 structural inconsistency regions were detected in the stitched image. These regions were output as distortion masks to provide target locations for subsequent local image reconstruction and compensation.

[0028] A local geometric model is established for each distortion location to analyze the deformation type (e.g., rotational deviation, distortion, stretching) of the region. Then, the expected pixel position offset is calculated based on the projected coordinates of the reference view image. Next, the system uses bilinear interpolation to resample these regions, i.e., by weighted averaging the grayscale or color values ​​of the four neighboring pixels to recalculate the pixel value of the distorted point. Bilinear interpolation is a reconstruction method with high smoothness, which can effectively alleviate image blockiness and jaggedness. In the experiment, the processing area size was controlled within 30×30 pixels to ensure local processing accuracy and avoid overall image distortion. To further optimize the overall consistency of the stitched image, the system also performs a full-image illumination balancing process after processing to adjust the brightness differences between images from different viewpoints. The output corrected panoramic stitched image has high visual consistency and structural integrity, and can be directly used for product recognition and price calculation model inference, significantly improving the accuracy and stability of the entire vending machine AI recognition system.

[0029] In this embodiment, see Figure 3 The diagram below illustrates the detailed implementation steps of step S2. In this embodiment, the detailed implementation steps of step S2 include: Real-time brightness detection and analysis of the internal lighting environment of the vending machine are performed to obtain real-time brightness detection data; Illumination distribution identification is performed on real-time brightness detection data to generate internal illumination distribution features; Based on the internal illumination distribution characteristics, a comparative analysis of regional brightness was conducted to mark areas with relatively dim illumination. Optimize the light source configuration for areas with low illumination, and acquire monitoring images of these areas after brightness enhancement. Based on the monitored images, the brightness of the corrected panoramic stitched images is enhanced and fused to construct a brightness-enhanced panoramic image. Multi-scale image noise filtering is performed on the brightness-enhanced panoramic image to obtain a filtered and optimized panoramic image.

[0030] In this embodiment, multiple cameras or dedicated light sensors (such as TSL2561, BH1750, etc.) deployed inside the vending machine periodically collect light intensity data. This data is then combined with the luminance components in the RGB image for calculation to obtain a more comprehensive light assessment. In the experimental setup, the camera sampling frequency is once every 5 seconds, and the image size is 640×480 pixels. The system converts the image to HSV or YCbCr color space and extracts the V or Y channel as the basic luminance data. Subsequently, the image is divided into several small regions according to a grid (e.g., 32×32 pixels), and the average luminance value and standard deviation of each region are calculated to generate a luminance histogram. By statistically analyzing the overall average luminance and distribution range of the image, it is determined whether the current lighting condition meets the standard (e.g., the average luminance should be higher than 120 / 255). The output is a real-time luminance detection dataset, containing the luminance value of each region, its trend, and whether it is within the effective luminance threshold range. The luminance data of the grid-divided regions from the previous step is remapped into the three-dimensional coordinate model of the vending machine to form a light heat map covering the vending machine's aisles. In this process, the system uses bilinear interpolation to estimate the brightness of unsampled areas to generate a continuous illumination distribution image. Then, image smoothing (using a 3×3 or 5×5 Gaussian filter) removes the influence of single-point brightness fluctuations, making the overall illumination distribution map more stable. Next, the system uses hierarchical clustering methods (such as Mean Shift or K-means) to divide different brightness regions into multiple levels, such as "high brightness area," "medium brightness area," and "low brightness area." In the experiment, the number of clusters was set to 3, and the brightness thresholds were approximately set to <100 (low brightness), 100~180 (medium brightness), and >180 (high brightness), forming an illumination distribution map with spatial labels. This distribution map, as an illumination feature model, can not only guide the configuration of light source compensation but also be used for illumination-adaptive image enhancement and recognition weight adjustment, improving the stability and reliability of the AI ​​system under uneven illumination.

[0031] The system calculates the average brightness of the entire image (e.g., the average of the V or Y channels), and then compares the local brightness of each grid region with the overall image brightness, defining a brightness deviation metric. If the brightness of a region is more than 30% lower than the average (e.g., the average brightness is 150, and the region is below 105), it is marked as a "darkly lit region." Furthermore, to avoid misjudgment, the system also uses the brightness standard deviation for structured judgment, eliminating local dark spots caused by object occlusion or color variations. In the experiment, a dual strategy of local-to-global ratio thresholding and brightness gradient variation was employed to ensure that the identified darkly lit regions possess stable spatial characteristics. The result is a binary mask image or label image, marking the locations of areas in the vending machine that require illumination compensation. These locations are often areas where product barcodes and text labels are concentrated; insufficient illumination directly affects the accuracy of the AI ​​recognition model, making this step highly important in the overall system. The configuration of the LED supplementary lights inside the vending machine will be dynamically adjusted based on the spatial location of the darkly lit regions to enhance the brightness of these areas. The light source optimization and driving module controls the brightness, current, and illumination angle of the LED module (supporting PWM dimming) to achieve targeted lighting of the target area. Based on the coordinates of the darker areas, the system assigns the supplementary lighting task to the nearest controllable light source unit, while considering the current ambient brightness and light source temperature rise limitations (typically limiting continuous LED brightness increase to no more than 80%) to balance the supplementary lighting effect and energy consumption. After completing the supplementary lighting driving action, the system waits 1-2 seconds for stable illumination, then re-captures the enhanced monitoring image via the camera and re-executes brightness detection. The newly acquired image must be compared with the previous image for brightness verification. If the brightness increase in the target area exceeds 25% and reaches the recommended recognition threshold (e.g., V channel > 130), the lighting compensation is considered effective. Otherwise, the system will perform further dimming optimization or issue an alarm.

[0032] Pixel-level spatial registration is performed between the panoramic image and the monitoring image to ensure consistent geometric correspondence between the enhanced regions in both images. Then, a linear weighted fusion method is applied to overlapping regions: the pixel values ​​of the enhanced image are used as the principal values ​​for dark areas, and a weighted average of the two is used for transition areas. To avoid unnatural edge transitions, the system uses 5×5 or 7×7 Gaussian convolution kernels at the fusion boundaries for smooth transition processing. Furthermore, full-image brightness normalization is performed to ensure that there are no brightness jumps or color temperature shifts in the enhanced regions. In experiments, this fusion process increased the brightness of dark areas by an average of 25%–40%, while simultaneously increasing the overall SSIM (Structural Similarity Index) by approximately 0.12, significantly enhancing image quality. The output brighter panoramic image serves as a new input source for the AI ​​recognition model, enabling products to be clearly captured and identified under various lighting conditions.

[0033] A multi-scale decomposition combined with edge-preserving filtering strategy was adopted. First, the panoramic image was decomposed into pyramids, breaking it down into multiple scale levels (e.g., 5 levels), which were then processed separately for low-frequency (background brightness), mid-frequency (structure and texture), and high-frequency (edge ​​details). For the low-frequency layers, mean filtering or bilateral filtering was used to smooth large-area brightness fluctuations. Gaussian-bilateral hybrid filtering was used for noise reduction in the mid-frequency layers, and guided filtering or nonlocal mean (NLM) was used for edge preservation in the high-frequency layers. In the experimental parameter settings, the Gaussian kernel size for the low-frequency layers was 9×9, for the mid-frequency layers it was 5×5, and for the high-frequency layers it was 21×21. After all processing was completed, the image was reconstructed, resulting in a filtered and optimized panoramic image with significantly improved visual quality, clear edges, balanced brightness, and extremely low noise. This image served as the input image for the AI ​​recognition module, significantly improving the accuracy of tasks such as product classification, text detection, and barcode recognition.

[0034] In this embodiment, reference Figure 4 The diagram below illustrates the detailed implementation steps of step S3. In this embodiment, the detailed implementation steps of step S3 include: Perform individual product visual recognition on the filtered and optimized panoramic image, and mark multiple product image frames within the vending machine; Pixel-level edge detection is performed on the multiple product image frames to extract the initial edge contour information of each product; Based on the initial edge contour information, edge contour position analysis is performed to extract the key edge contour point coordinate sequence; Centroid calculation is performed based on the coordinate sequence of key edge contour points to obtain pixel-level center positioning points; Define a unique code for each product, perform spatial product distribution analysis based on pixel-level center positioning points, and construct a three-dimensional product distribution model based on the unique code.

[0035] In this embodiment, a visual recognition task is performed for each item, aiming to accurately detect all visible items in the image and label them using bounding boxes. A pre-trained deep convolutional neural network model (such as YOLOv5, EfficientDet, or Faster R-CNN) is used for multi-class object detection. The model has been trained on a large number of vending machine items and is capable of recognizing items with different shapes, packaging, and colors. The system inputs the panoramic image into the detection network according to its resolution (e.g., 2048×1536 pixels). The model output includes the category label, confidence score, and bounding box coordinates for each item. To ensure detection accuracy, a confidence threshold (e.g., 0.6) is set to filter low-quality prediction results, and non-maximum suppression (NMS) is performed on overlapping bounding boxes to prevent duplicate labeling. In the experimental scenario, an average of 35-50 items can be recognized per image, with a recognition accuracy (mAP) of over 0.87. This results in a preliminary annotation result containing the image bounding box positions and categories of all items. Pixel-level edge detection is performed on the internal region of each product image frame to accurately extract the initial contour information of the product. This process mainly uses classic image processing algorithms, such as Canny edge detection or the gradient-direction-based Sobel operator, combined with the grayscale variation characteristics of the product image to perform edge extraction. Grayscale conversion and Gaussian blur processing (Gaussian kernel size is typically 5×5, sigma is 1.4) are applied to the product image region to suppress image noise; then, double-threshold edge detection is performed, typically using a lower threshold of 50~150 to capture more complete edge lines. After edge extraction, connected edge segments are identified using a contour detection function (such as findContours in OpenCV), and multiple contours are filtered based on area and edge closure, retaining only the main contour as the initial boundary shape of the product. In the experiment, an average of 35 candidate contours were extracted for each product, and the optimal edge information was selected through shape similarity matching and centroid distribution verification.

[0036] The complete edge contours are sorted clockwise to form a closed point set path. Then, a contour simplification algorithm (such as the Douglas-Peucker algorithm) is applied to approximate the edge points with curves to retain the main shape inflection points. This algorithm balances the degree of simplification with contour fidelity by controlling the maximum error threshold (e.g., 2-3 pixels). In experiments, each product can be compressed from the original hundreds of edge points to an average of 20-50 key points. Subsequently, the system further analyzes the distribution of corner points and curvature changes of the contour, and retains key corners and edge peak points with weights to form a stable sequence of edge points. Each point is recorded as a two-dimensional coordinate (x, y), along with the point's local normal vector information, which facilitates subsequent spatial modeling and docking with 3D structures. This key point coordinate sequence is not only used for centroid calculation, but also for subsequent rotation posture judgment and region segmentation of the product, providing a high-precision description of the product's shape for the AI ​​post-processing module.

[0037] After extracting the key edge contour points of the product, the centroid of the product, i.e., the pixel-level center positioning point, is calculated based on the geometric relationship of these points. The centroid calculation method is mainly based on the centroid formula of a two-dimensional polygon: the closed edge contour points are arranged in sequence to form a polygon, and the center position in the x and y directions is obtained by performing an area-weighted average of the coordinates of all points. The system considers the triangular region formed between all key points and calculates the centroid coordinates (Cx, Cy) by combining the area and vertex coordinates. In special cases, such as irregular objects or non-closed edges, a mass center calculation method based on pixel connected domains can also be used. That is, after binarizing the product pixel region, the center coordinates of all foreground pixels are counted as the centroid. In the experiment, the accuracy of the centroid coordinates of each product is controlled within ±2 pixels, and the deviation from the actual physical center of the product is less than 5mm (verified by depth map). The calculated centroid points are not only used for product coding and spatial analysis, but also serve as a bridge between image recognition results and real-world position calibration, supporting high-precision vending machine product positioning and inventory statistics functions.

[0038] Each identified product instance is assigned a unique code, and a 3D spatial distribution model is constructed based on its centroid location point. The unique code typically uses a standard structure format, containing fields such as image ID, product category ID, centroid coordinates, and recognition confidence level, for example, the format: [IMG20250804_35_CL012_X512_Y278_CONF0.94]. Based on the position of each centroid point in the panoramic image, combined with the camera's intrinsic parameters and depth map data, the system reconstructs the product in a 3D coordinate system. If an RGB-D camera (such as RealSense) is used, the centroid pixel coordinates can be directly combined with the depth value to convert it into 3D coordinates (X, Y, Z), forming the point cloud center. If a multi-angle image fusion scheme is used, spatial reconstruction is performed from centroid points of multiple perspectives using triangulation. The 3D coordinates of all products are normalized and mapped to the vending machine's internal coordinate model (unit: cm), and categorized according to shelf level. The system outputs a 3D distribution model of the goods, which can be used for monitoring the arrangement status of goods and issuing early warnings of out-of-stock items. It can also be linked with the selling price information to achieve dynamic price recognition and intelligent recommendation based on spatial location, which is an important support for the core functions of the entire intelligent vending system.

[0039] In this embodiment, step S4 includes the following steps: The system identifies the vending machine's door opening signal, triggering a high-frequency image acquisition mode. When the vending machine's door closing signal is triggered, it acquires the latest internal detection image. Spatial alignment matching is performed on the internal detection image to obtain a spatially aligned image; Semantic segmentation is performed on spatially aligned images to extract product-level semantic representations; Based on commodity-level semantic representation, a differential comparison calculation of door opening and closing is performed on a 3D commodity distribution model to obtain semantically changed image regions. The vending machine's real-time missing items are determined based on semantic-level changes in image regions; the real-time missing items are located in 3D and similarity recognition is performed to obtain a list of missing items.

[0040] In this embodiment, the system monitors changes in the door control status in real time to determine whether a user is retrieving or placing goods. The vending machine's door control signal is detected by a built-in magnetic sensor or Hall effect sensor. The system polls the door control status every 10ms. When the door is detected to move from a closed state to an open state, it is considered an "opening event." At this time, the system immediately switches to a high-frequency image acquisition mode, calling a preset camera array (generally including an RGB camera and a depth camera) to continuously capture images at a frequency of 30 frames per second or higher. During acquisition, the exposure time is appropriately shortened (e.g., set to within 10ms) to avoid motion blur. When a "closing event" signal is detected (i.e., the door closes again after opening), the system immediately terminates the high-frequency capture and selects the image after the door closes as the "latest detected image." This image is used for differential comparison with the model before the door closes. In the experimental environment, the response delay of this triggering mechanism is controlled within 50ms, ensuring high synchronization between user operation and image acquisition, effectively capturing the image state before and after goods changes, laying the foundation for goods missing recognition. After acquiring images during the door opening and closing cycle, the system spatially aligns the image acquired at the closing moment with the original panoramic image to construct a consistent coordinate reference frame and enable inter-image differential analysis. Spatial alignment employs an image registration algorithm based on feature point matching and geometric transformation. Specifically, SIFT or ORB feature points are extracted from the current detection image and matched with feature points in the original reference image. A sufficient number of evenly distributed point pairs (typically more than 300 pairs) are selected. Then, the RANSAC algorithm is used to estimate the affine or perspective transformation matrix to obtain the rotation, translation, and scaling parameters between the images. After mapping the current image to the coordinate system of the original image, image resampling and boundary alignment are performed to ensure pixel-level consistency between the two images. In the experimental setup, the system controls the registration error to within 1.5 pixels and evaluates the registration quality using Structural Similarity (SSIM), requiring a score higher than 0.9. The generated spatially aligned image maintains geometric consistency with the original model and can be used for subsequent semantic-level differential comparison and change region detection.

[0041] To further clarify the semantic information of the products corresponding to each region in the image, the system needs to perform semantic segmentation on the spatially aligned detection image. Semantic segmentation aims to classify each pixel in the image into a specific product category or background category. Commonly used models include DeepLabv3+, SegFormer, or U-Net structures. These models have been pre-trained on labeled datasets containing various product forms and have the ability to recognize different product packaging, sizes, and colors. The system feeds the aligned image into the segmentation model in standard-sized slices (e.g., 512×512 or 1024×768), outputting a category label mapping map for each pixel. In the resulting image, each product region is assigned a unique semantic category code, forming a connected component partition. To improve accuracy, a CRF (Conditional Random Field) post-processing method is also combined to optimize pixel labels in edge regions, reducing classification fragmentation or misclassification. In the experiment, the semantic segmentation mIoU value of each frame remained above 0.82, effectively distinguishing complex situations such as product boundaries, empty storage locations, and transparent occlusions. After obtaining the semantic segmentation results of the current image, the system performs a differential comparison with the original 3D product distribution model to identify changes in products caused by door opening and closing operations. This comparison is based on two levels: first, a 2D semantic map comparison, where the system performs pixel-level difference analysis on the semantic masks in the detected image and the original image to identify areas where semantic labels have changed (e.g., areas that originally belonged to products are replaced by background); second, a 3D model comparison, where the system queries the 3D product distribution database based on the product ID and centroid position corresponding to the semantic label to check if the product instance exists in the current frame. If a previously marked product no longer appears, or the corresponding area becomes blank, it is determined that the product has been removed by the user. In actual testing, an average of 2-4 difference areas were detected per door opening and closing event. The system reduces false alarms caused by occlusion or lighting changes through cross-validation of location information and semantic categories. The output is a semantic-level change area map, indicating the location, change category, and confidence score of the product change.

[0042] After completing semantic differential analysis, the system compares and confirms each semantically changed region to determine if any items are missing from the vending machine and outputs a list of corresponding item IDs. The judgment is based on two main conditions: first, the changed region spatially overlaps with the original item location; second, the semantic label changes from "item type" to "background type" or "shelf type." When both conditions are met, the system marks the item as "removed." If multiple item locations change simultaneously, the system can combine temporal information to determine the user's removal order and potential occlusion effects. In experimental verification, the item missing detection accuracy is higher than 94%, and a complete missing item list can be output within approximately one second after the user closes the machine. This result is also synchronously updated to the system database for subsequent item replenishment reminders, inventory monitoring, and sales record generation, ensuring intelligent and transparent vending machine management. After confirming missing items, the system further uses a 3D item model to reconstruct the original spatial coordinates of the missing items for accurate 3D positioning and performs semantic analysis on the category of the missing item and its substitutes. The 3D localization process, based on the original centroid coordinates and camera intrinsic parameters, backprojects the centroid of the missing product's 2D image into a 3D point (X, Y, Z), with a localization error controlled within ±1.5cm. Building upon the 3D coordinates, the system also incorporates product category tags for similar product identification, searching for other product instances of the same category, brand, or series on the current shelf. This identification process uses fuzzy matching based on the product's semantic coding structure (e.g., brand-type-packaging), considering both product appearance similarity and shelf proximity to assess whether the product has been replaced rather than is missing. In the experimental system, the product similarity matching accuracy reached over 91%. The system outputs a "Product Change List," including fields such as the missing product ID, location, category, possible substitutes, and operation time, which serves as the input for the AI ​​price recognition module, ensuring that sales data and inventory status are updated synchronously, enabling full-process tracking of every product interaction.

[0043] In this embodiment, step S5 includes the following steps: The product code is parsed from the product change list to extract the product SKU identification information; Based on the product SKU identification information, a local database query and matching is performed to generate price information for missing products; The missing product price information is used to perform parallel calculations on multiple prices to obtain the price calculation results; Cross-validation of weight sensor data is performed based on the price calculation results to obtain the validation results; The confidence level of the verification results is assessed to obtain the confidence level of the price results; When the confidence level of the price result is lower than the preset confidence level threshold, it is submitted to the cloud for manual review and processing; when the confidence level of the price result is higher than the preset confidence level threshold, a price list is output. The system performs online iterative learning based on the price list, optimizes price calculations, and completes parallel price calculation and verification tasks.

[0044] In this embodiment, after detecting changes in goods and generating a change list, the system needs to parse the code of each goods instance in the change list to extract a unique product identifier, i.e., SKU (Stock Keeping Unit). The SKU, as a unique identifier for a product in the vending machine, is typically composed of multiple dimensions such as the product's brand, type, packaging specifications, size, and location, forming a structured code, such as "BRD101-CAN330ML-SHLF02". The system extracts the coded content corresponding to the coordinates of each product change by consulting the 3D product distribution model and the product metadata mapping table, and parses the SKU components according to preset rules. This process must ensure the consistency between the encoding and the semantic information of image recognition to avoid encoding errors caused by image overlap or occlusion. In the test system, the encoding parsing accuracy reached 99.2%, mainly relying on the standardized product codebook and the model's synchronous update mechanism. After completing the SKU parsing, the system binds a unique identifier to each missing product to obtain the SKU of each missing product. The system then uses this as a key index to initiate a query operation in the local price database. The database structure uses the product SKU as the primary key, associated with fields such as product name, specifications, price, promotional discounts, and supplier, supporting multi-SKU price queries. The system first performs exact matching based on the SKU. If the SKUs match perfectly, the corresponding price is retrieved directly. If the SKU prefix matches but the specifications are inconsistent (e.g., capacity differences), the system calls a fuzzy matching function to return a price range for candidate products based on product similarity (using Jaccard or vector cosine similarity scoring). In the experimental environment, the local product database capacity is controlled within 10,000 records, and the query latency is less than 20ms, essentially achieving real-time price response. To cope with promotional price fluctuations, the system also has a price version control mechanism to ensure that query results are based on the latest price strategy table. All successfully matched missing products are assigned corresponding unit price information and summarized into a price item record table by SKU.

[0045] After obtaining the basic price information of the missing product, the system needs to perform parallel calculations of the price to accurately reflect the user's actual purchase amount. The core of multi-price parallel calculation lies in the fact that for each missing product SKU, it not only calculates its standard selling price but also simultaneously handles different price paths such as possible discount prices, member prices, and bundled promotional prices. The system constructs a price decision tree model for this purpose, reasoning according to the logical flow of "product identity → applicable discount rules → settlement path". For example, when an SKU matches a buy-one-get-one-free promotion (such as "second item half price"), the system automatically determines whether it applies to the current product bundle and returns a weighted average price. The entire calculation process is executed through multi-threading, using a GPU parallel architecture, with a processing latency of no more than 50ms for batches of products (e.g., 5-10 items). The calculation result will output the calculated price per product, along with metadata such as the calculation path, usage rules, and price source. This price calculation result serves as the preliminary settlement price and will undergo sensor verification and confidence assessment to ensure accurate pricing even in complex discount scenarios. To improve the reliability of product recognition and price calculation, the system introduces weight sensor data as a cross-validation method, comparing the total weight of identified missing products with the actual weighing data. Each shelf inside the vending machine is equipped with a high-precision strain gauge weight sensor with a sampling accuracy of ±1g and a sampling frequency of 20Hz, providing real-time feedback on weight changes on the shelves. Each time the door closes, the system obtains the total weight change of the current shelf and adds it to the predefined weight (from the SKU table) of each product in the missing product list determined by image recognition, deriving the theoretical missing weight. By comparing the difference (Δw) between the theoretical value and the sensor's measured value, the system judges the accuracy of the recognition. A tolerance threshold of ±3g is set; if Δw exceeds this range, it is marked as having potential recognition bias. In experimental scenarios, this method achieves a verification accuracy of 96.7%, performing particularly well in distinguishing products that look similar but have different weights. The cross-validation results serve as one of the core parameters for subsequent confidence scoring, effectively strengthening the physical empirical foundation of the image recognition system.

[0046] After completing image recognition, SKU matching, price calculation, and sensor cross-validation, the system fuses the results from various dimensions to calculate the price confidence score for the entire transaction. The scoring model is built based on a weighted logistic regression method, mainly considering the following factors: 1) Image recognition confidence (classification confidence output based on the image segmentation model, weight 0.4); 2) SKU matching similarity (string matching accuracy, weight 0.2); 3) Sensor validation error (relative distance of Δw from the standard range, weight 0.3); 4) Price calculation consistency (whether it is a unique decision path, weight 0.1). The confidence score is output after weighting each factor. The system sets a default confidence threshold of 0.85. If the score is higher than this value, it is considered "reliable price recognition"; if it is lower than this value, manual review is required. Two processing paths are then used based on the confidence score results. When the confidence level of the price result is lower than a set threshold (e.g., 0.85), the system automatically packages and uploads all metadata related to the recognition and calculation process, including images, price paths, recognition logs, and sensor data, to a cloud-based manual review platform. A manual annotation team then reviews the product changes, SKU matching, and pricing decisions, correcting recognition errors and providing model feedback. Cloud-based review typically completes within 30 seconds and supports a feedback mechanism to improve the training dataset and enhance model robustness. Conversely, when the confidence level is higher than the threshold, the system directly outputs a price list, detailing the SKU, product name, quantity, unit price, discount path, and total price, and simultaneously sends it to the payment system for settlement. In experimental verification, the average response time for high-confidence transactions was 0.8 seconds, while the average response time for low-confidence review processing was 15-30 seconds. This mechanism ensures both efficiency and accuracy, guaranteeing that the vending machine can still provide stable and accurate automated vending services in unattended scenarios.

[0047] In this embodiment, the specific steps for performing online iterative learning based on the price list, optimizing price calculations, and completing the parallel price calculation and verification task are as follows: The price list is visualized via a user interface, and user purchase confirmations are recognized. Once the user's purchase confirmation result is confirmed, the user is taken to the payment interface. The system checks if the user has completed the payment and then completes the sales price calculation. When a user's purchase confirmation result is abnormal, it is uploaded to the cloud for manual review and processing, and then tagged to generate a tag training dataset. The label training dataset is used for online iterative learning, and price calculation is optimized to complete the parallel price calculation and verification task.

[0048] In this embodiment, the price list is presented to the user for confirmation through an interactive interface. The interface design uses a touch-screen LCD or an embedded transparent OLED display module, supporting side-by-side display of text and images. Each item is presented with an image frame, SKU name, quantity, unit price, and subtotal. The system also dynamically renders product thumbnails based on image recognition marker information to enhance user intuitiveness. To prevent user misoperation, the interface provides interactive buttons such as "Confirm Purchase," "Modify List," and "Call Customer Service." The interactive area is located in the lower center of the screen, with a response latency controlled within 100ms to ensure smooth operation. Users complete the confirmation action through physical touch, facial expression recognition, or voice commands. Facial confirmation uses a ResNet50+Softmax classification model to perform binary classification of "nodding" and "shaking" actions, achieving an experimental recognition accuracy of 95.6%. The system captures user feedback in real time. If the recognition result is confirmation, the payment process is triggered; if rejection or abnormal behavior is detected, the system records the operation log and uploads it to the exception handling module. After the user confirms the price list, the system automatically jumps to the payment interface, initiates the payment process, and completes the settlement stage of the entire sales process. The payment interface is seamlessly integrated with third-party payment platforms (such as WeChat, Alipay, and bank card NFC) through an embedded module in the vending machine. Users can complete payment through various methods such as scanning QR codes, facial recognition, card swiping, or fingerprint recognition. The system dynamically generates payment order codes using a QR code generation engine (such as the ZXing framework), combining the product SKU, price details, vending machine ID, and timestamp to generate a unique transaction record, ensuring the uniqueness and traceability of the payment path. The payment process involves verification communication with the payment gateway through local middleware, typically returning a success or failure status within 0.5 seconds. After payment, the system synchronously updates the corresponding product inventory status and records complete log data for the transaction (including payment ID, SKU details, payment method, and time). This stage marks the end of the entire sales price calculation process and forms a closed-loop transaction data structure for further sales statistics and anomaly retrieval analysis.

[0049] All relevant data, including the identified images, product change lists, price details, user interaction records, and sensor data, were packaged and uploaded to a cloud-based human review platform. The uploaded data consisted of structured logs and visual image pairs, with each anomaly event assigned a unique anomaly ID for easy labeling and tracing. Human reviewers on the cloud platform verified misidentified products or incorrect prices based on the images and transaction information. Human feedback was labeled using standard tags, such as "misidentified product," "missed product," and "pricing decision error," with detailed annotations on product frame positions, SKU codes, and price paths. This labeled data, along with the images, constituted standard training samples and were uniformly stored in a "labeled training dataset" for subsequent model training and performance calibration. In a pilot project, the daily average occurrence rate of anomalies was approximately 2.1%, while the model's misidentification rate decreased by an average of approximately 3.8% within a week through the cloud-based labeling mechanism, validating the crucial role of the human intervention + automatic learning model in system self-optimization. The system employs an image object detection architecture based on deep convolutional networks (such as YOLOv5 and EfficientDet) and introduces new labeled data through transfer learning to achieve incremental model learning without complete retraining. For price decision paths, the system optimizes the price calculation logic tree, introduces anomaly-labeled samples to construct a reinforcement learning model, and scores different discount combinations and product combinations to correct ambiguous price path decisions. During iterative training, the system introduces a dynamic learning rate (e.g., an initial learning rate of 1e-4 with a decay factor of 0.95) and uses a validation set to monitor the model's generalization performance in real time. Training results are synchronously updated to the local model cache, with a hot model update every 24 hours to ensure that the online recognition logic remains consistent with the data. The optimization goal is to improve the overall accuracy and response efficiency of the system in image recognition, SKU matching, and price path determination. In practice, after 7 consecutive days of online learning, the overall recognition-computation accuracy of the system increased to 98.9%, significantly reducing the frequency of transaction interventions caused by user-confirmed anomalies and improving the intelligent decision-making level and autonomous closed-loop capability of the automated vending system.

[0050] In this embodiment, an AI-based visual recognition-based vending machine multi-item sales price calculation system is provided, used to execute the AI-based visual recognition-based vending machine multi-item sales price calculation method described above, including: The panoramic stitching module is used to acquire a sequence of three-dimensional images inside the vending machine, stitch the panoramic images at the edges, and obtain a corrected panoramic stitched image. The image optimization module is used to perform multi-scale image noise filtering on the corrected panoramic stitched image to obtain a filtered and optimized panoramic image. The product distribution module is used to perform visual recognition of each product in the filtered and optimized panoramic image, analyze the spatial distribution of products, and build a three-dimensional product distribution model. The differential comparison module is used to identify the vending machine's door opening signal, collect the latest internal detection images, and perform differential comparison analysis on the three-dimensional product distribution model to obtain a list of product changes. The parallel computing module is used to perform parallel calculations on multiple prices for the product change list and output a price list.

[0051] Therefore, the embodiments should be considered as exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of the equivalents of the application are intended to be included within the invention.

[0052] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein are implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features of the invention herein.

Claims

1. A method for calculating the selling price of multiple items in a vending machine based on AI visual recognition, characterized in that, Includes the following steps: Step S1: Collect a three-dimensional image sequence inside the vending machine, perform panoramic image edge stitching, and obtain a corrected panoramic stitched image; Step S2: Perform multi-scale image noise filtering on the corrected panoramic stitched image to obtain a filtered and optimized panoramic image; Step S3: Perform visual recognition of each product in the filtered and optimized panoramic image, analyze the spatial distribution of products, and construct a three-dimensional product distribution model; Step S4: Identify the vending machine's door opening signal, collect the latest internal detection images, and perform differential comparison analysis on the three-dimensional product distribution model to obtain a list of product changes; Step S5: Perform parallel calculations on multiple prices for the product change list and output the price list.

2. The method for calculating the selling price of multiple items in a vending machine based on AI visual recognition according to claim 1, characterized in that, The specific steps of step S1 are as follows: Multi-angle stereoscopic imaging of the internal three-dimensional space of the vending machine is performed to acquire a sequence of three-dimensional images. Image feature point recognition is performed on a 3D image sequence, and image feature points from multiple angles are marked; Multi-angle visual registration is performed based on image feature points from multiple angles to obtain multi-angle visual registration parameters; Image overlapping region detection and extraction are performed based on multi-angle visual registration parameters; Calculate the geometric transformation parameters of the overlapping region to obtain the rotation, translation, and scaling relationships between the images; The three-dimensional image sequence is edge-stitched according to the rotation, translation and scaling relationships to obtain a corrected panoramic stitched image.

3. The method for calculating the selling price of multiple items in a vending machine based on AI visual recognition according to claim 2, characterized in that, The specific steps for performing edge stitching processing on the 3D image sequence based on the rotation, translation, and scaling relationships to obtain the corrected panoramic stitched image are as follows: Based on the rotation, translation, and scaling relationships, multi-angle image edge detection is performed on the three-dimensional image sequence to identify the angular boundaries between different images; Calculate the pixel gradient at the angular boundary to determine the image abrupt stitching seam; Morphological refinement of the stitching seams in the image abrupt changes yields stitching edge lines with a width of one pixel. Pixel-level point-by-point calculations are performed on the spliced ​​edge lines to obtain pixel-level coordinate indices; Multi-band image stitching is performed based on pixel-level coordinate indexing to obtain a three-dimensional stitched image; Perform splicing geometric distortion identification on 3D stitched images and mark the locations of geometric distortions; Local geometric correction is performed on the locations of geometric distortions, and bilinear interpolation resampling is used to construct a corrected panoramic stitched image.

4. The method for calculating the selling price of multiple items in a vending machine based on AI visual recognition according to claim 1, characterized in that, The specific steps of step S2 are as follows: Real-time brightness detection and analysis of the internal lighting environment of the vending machine are performed to obtain real-time brightness detection data; Illumination distribution identification is performed on real-time brightness detection data to generate internal illumination distribution features; Based on the internal illumination distribution characteristics, a comparative analysis of regional brightness was conducted to mark areas with relatively dim illumination. Optimize the light source configuration for areas with low illumination, and acquire monitoring images of these areas after brightness enhancement. Based on the monitored images, the brightness of the corrected panoramic stitched images is enhanced and fused to construct a brightness-enhanced panoramic image. Multi-scale image noise filtering is performed on the brightness-enhanced panoramic image to obtain a filtered and optimized panoramic image.

5. The method for calculating the selling price of multiple items in a vending machine based on AI visual recognition according to claim 1, characterized in that, Step S3 is as follows: Perform individual product visual recognition on the filtered and optimized panoramic image, and mark multiple product image frames within the vending machine; Pixel-level edge detection is performed on the multiple product image frames to extract the initial edge contour information of each product; Based on the initial edge contour information, edge contour position analysis is performed to extract the key edge contour point coordinate sequence; Centroid calculation is performed based on the coordinate sequence of key edge contour points to obtain pixel-level center positioning points; Define a unique code for each product, perform spatial product distribution analysis based on pixel-level center positioning points, and construct a three-dimensional product distribution model based on the unique code.

6. The method for calculating the selling price of multiple items in a vending machine based on AI visual recognition according to claim 1, characterized in that, The specific steps of step S4 are as follows: The system identifies the vending machine's door opening signal, triggering a high-frequency image acquisition mode. When the vending machine's door closing signal is triggered, it acquires the latest internal detection image. Spatial alignment matching is performed on the internal detection image to obtain a spatially aligned image; Semantic segmentation is performed on spatially aligned images to extract product-level semantic representations; Based on commodity-level semantic representation, a differential comparison calculation of door opening and closing is performed on a 3D commodity distribution model to obtain semantically changed image regions. The vending machine's real-time missing items are determined based on semantic-level changes in image regions; the real-time missing items are located in 3D and similarity recognition is performed to obtain a list of missing items.

7. The method for calculating the selling price of multiple items in a vending machine based on AI visual recognition according to claim 1, characterized in that, The specific steps of step S5 are as follows: The product code is parsed from the product change list to extract the product SKU identification information; Based on the product SKU identification information, a local database query and matching is performed to generate price information for missing products; The missing product price information is used to perform parallel calculations on multiple prices to obtain the price calculation results; Cross-validation of weight sensor data is performed based on the price calculation results to obtain the validation results; The confidence level of the verification results is assessed to obtain the confidence level of the price results; When the confidence level of the price result is lower than the preset confidence level threshold, it is submitted to the cloud for manual review and processing. When the confidence level of the price result is higher than the preset confidence level threshold, output the price list; The system performs online iterative learning based on the price list, optimizes price calculations, and completes parallel price calculation and verification tasks.

8. The method for calculating the selling price of multiple items in a vending machine based on AI visual recognition according to claim 7, characterized in that, The specific steps for performing online iterative learning based on the price list, optimizing price calculations, and completing the parallel price calculation and verification task are as follows: The price list is visualized via a user interface, and user purchase confirmations are recognized. Once the user's purchase confirmation result is confirmed, the user is taken to the payment interface. The system checks if the user has completed the payment and then completes the sales price calculation. When a user's purchase confirmation result is abnormal, it is uploaded to the cloud for manual review and processing, and then tagged to generate a tag training dataset. The label training dataset is used for online iterative learning, and price calculation is optimized to complete the parallel price calculation and verification task.

9. A vending machine multi-item sales price calculation system based on AI visual recognition, characterized in that, The method for calculating the multi-item sales price of a vending machine based on AI visual recognition as described in claim 1 includes: The panoramic stitching module is used to acquire a sequence of three-dimensional images inside the vending machine, stitch the panoramic images at the edges, and obtain a corrected panoramic stitched image. The image optimization module is used to perform multi-scale image noise filtering on the corrected panoramic stitched image to obtain a filtered and optimized panoramic image. The product distribution module is used to perform visual recognition of each product in the filtered and optimized panoramic image, analyze the spatial distribution of products, and build a three-dimensional product distribution model. The differential comparison module is used to identify the vending machine's door opening signal, collect the latest internal detection images, and perform differential comparison analysis on the three-dimensional product distribution model to obtain a list of product changes. The parallel computing module is used to perform parallel calculations on multiple prices for the product change list and output a price list.

Images