AI Vision-Based Item Recognition System and Method for Vending Machines
By generating a transparent packaging rotation sequence and refraction interference trajectory, reconstructing the contour, and adjusting the acquisition rhythm, the problem of recognition failure in transparent packaging item recognition was solved, and the stability and accuracy of item recognition in vending machines were achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HANGZHOU CUIYING DATA TECHNOLOGY CO LTD
- Filing Date
- 2026-04-23
- Publication Date
- 2026-07-14
Smart Images
Figure CN122391668A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of intelligent retail identification technology, specifically to an AI vision-based unmanned vending machine item identification system and method. Background Technology
[0002] AI-based vision-based item recognition in vending machines refers to the process of continuously acquiring images of the vending machine's product aisles, dispensing slots, and user operation areas during its automated operation. Using artificial intelligence vision methods, the machine analyzes the shape, size, texture, color distribution, and positional relationships of items within the images. Based on the analysis results, it determines the type, quantity, and placement status of the items, thus completing pre-sale display interpretation, product retrieval confirmation, restocking verification, and abnormal behavior identification. This process uses continuous images as input and employs frame-by-frame analysis as the main approach. By comparing spatial changes and feature variations between consecutive images, the vending machine can automatically identify and manage items without human intervention.
[0003] The existing technology has the following shortcomings: Current technologies for visual recognition of transparently packaged items in vending machines often rely on edge textures and brightness variations to determine the item's extent. However, transparent packaging is prone to strong refraction when rotating, creating bright lines that form clear outlines and briefly cover the edges of the actual object. Because the brightness and shape of these bright lines can be easily misinterpreted by the algorithm as the object's outer contour, the obscured areas are treated as blank areas, leading to truncated object extent, missing texture information, and feature comparison failure. Further recognition may result in the algorithm judging the object as incomplete, moved, or unidentifiable, or even misjudging its non-existence, causing the entire recognition process to fail. These problems are more easily triggered when transparent packaging rotates rapidly, twists locally, or experiences sudden changes in light angle, making it difficult for current technologies to guarantee stable and reliable recognition results in dynamic scenarios, forming a significant technical bottleneck in the item recognition process of vending machines.
[0004] The information disclosed in the background section is only intended to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention
[0005] The purpose of this invention is to provide an AI vision-based item recognition system and method for vending machines to solve the problems mentioned in the background art.
[0006] To achieve the above objectives, the present invention provides the following technical solution: an AI vision-based method for item recognition in vending machines, comprising the following steps: Collect continuous footage of the transparent packaging in the vending machine during the picking and placing process, and unify the rotation changes in the continuous footage to the same time rhythm to generate a transparent packaging rotation sequence; Track the movement trajectory of the refracted bright line in the continuous image during the transparent packaging rotation sequence, and generate a refraction interference trajectory draft based on the time position of the refracted bright line covering the edge of the transparent packaging. Extract edge segments that are not obscured by refracted bright lines from the images before and after the coverage of the refracted interference trajectory record, and continuously stitch the edge segments to reconstruct the transparent packaging outline, thereby generating a candidate set of transparent packaging outlines; The real-time recognition results are compared using a candidate set of transparent packaging contours, and a time record of misjudgment risk is generated based on the contour contraction position and contour breakage position. The acquisition rhythm and lighting sequence of continuous images are adjusted according to the time record of misjudgment risk. The analysis operation is paused at the time position of misjudgment risk and reverse dark pattern compensation is added. The refraction interference of the real-time recognition result is corrected to output the final recognition result.
[0007] Preferably, the steps for generating the transparent packaging rotation sequence are as follows: The system captures continuous footage of transparent packaging during the handling and placement process. The continuous footage covers the entire process of the transparent packaging being stationary, in contact, rotating, tilting, swinging, and moving, and forms a continuous sequence of footage at fixed time intervals. The rotation and movement of transparent packaging in a continuous sequence of images are organized sequentially, and the continuous images are arranged according to the order of posture changes to form a sequence of images reflecting the process of taking out and putting in transparent packaging. The timing rhythm of consecutive shots is adjusted according to the order of the shots, and the consecutive shots are redistributed according to a unified time step to form a shot arrangement with a consistent timing rhythm. The continuous images with adjusted timing are combined in chronological order to generate a transparent packaging rotation sequence, which is used to express the continuous posture changes of the transparent packaging during the handling process.
[0008] Preferably, the steps for generating the refraction interference trajectory are as follows: Locate the area in which the refracted bright line appears in the continuous frame of the transparent packaging rotation sequence, and record the position and shape of the refracted bright line in each frame. Based on the records of refracted bright lines in each frame, the correlation processing is performed according to the time sequence of the transparent packaging rotation sequence to form the movement trajectory of the refracted bright lines in the continuous frames. The movement trajectory of the refracted bright line is correlated with the outline position of the transparent packaging edge in each frame, and the time position when the refracted bright line covers the transparent packaging edge is recorded. The movement trajectory of the refracted bright line is integrated with the time position of covering the edge of the transparent packaging to form a refracted interference trajectory draft, which is used to characterize the interference of the refracted bright line during the picking and placing process.
[0009] Preferably, when recording the time position of the refracted bright line covering the edge of the transparent packaging, the coverage area and duration of the refracted bright line in the corresponding image are recorded simultaneously, and written into the refractive interference trajectory file according to the time sequence of the transparent packaging rotation sequence, so as to distinguish the changes in the intensity of refractive interference at different time positions.
[0010] Preferably, the steps for generating the transparent packaging outline candidate set are as follows: In the coverage time position marked by the refraction interference trajectory, distinguish between the continuous images before the refraction bright line coverage behavior occurs and the continuous images after the coverage behavior ends, and obtain the images before and after coverage. Extract the edge lines of the transparent packaging at the corresponding time positions in the images before and after the overlay, forming edge segments that reflect the changes in the posture of the transparent packaging. According to the time sequence of the transparent packaging rotation, the position, extension direction and posture changes of the edge segments in the picture are sorted out, and the edge segments are arranged according to the time sequence. Spatial alignment and continuous connection of edge segments arranged in chronological order are performed to reconstruct the outer contour of the transparent packaging, and the resulting outer contour is organized into a candidate set of transparent packaging contours.
[0011] Preferably, in the process of spatial alignment and continuous connection of edge segments, the edge segments at adjacent time positions are correlated according to the orientation change direction of the transparent packaging in the rotation sequence, and the edge segments are continuously connected in combination with the extension direction and curvature change relationship of the edge segments in the picture, so as to form an outer contour that matches the picking and putting action of the transparent packaging, and limit the candidate set of transparent packaging contours to reflect the continuous shape change of the transparent packaging under different postures.
[0012] Preferably, the steps for generating the misjudgment risk time record are as follows: Record the outline shape of the transparent packaging in continuous images during real-time recognition, forming a real-time outline record corresponding to each frame; Based on the comparison between real-time contour records and transparent packaging contour candidate sets, the differences between real-time contours and candidate contours in terms of extension direction, curve direction and positional changes are analyzed. During the comparison process, the contour contraction and contour breakage locations that appear in the real-time contour are sorted out, and the contour contraction and contour breakage locations are associated with the time positions of the transparent packaging rotation sequence. The contour shrinkage and contour breakage positions are arranged according to the time sequence of the transparent packaging rotation sequence to form a time record of misjudgment risk that characterizes real-time risk identification.
[0013] Preferably, when forming the misjudgment risk time record, the time intervals corresponding to the contour contraction position and the time intervals corresponding to the contour breakage position are marked respectively, and the time distribution of different types of contour anomalies is distinguished in the misjudgment risk time record to indicate the risk source of contour changes during real-time identification.
[0014] Preferably, the acquisition rhythm and lighting sequence of continuous images are adjusted according to the time record of misjudgment risk. The analysis operation is paused at the corresponding time position and reverse dark pattern compensation is added. The steps for correcting refraction interference in the real-time recognition results are as follows: Based on the time record of misjudgment risk, the time position of continuous images is determined, and the acquisition rhythm of continuous images is adjusted at the corresponding time position to form a densely acquired sequence of continuous images. Based on the adjusted continuous image sequence, the order of supplementary lighting is changed at the time position corresponding to the time of misjudgment risk, and the light incident direction of the transparent packaging at the corresponding time position is adjusted. At the time points when the acquisition rhythm and fill light sequence are adjusted, pause the analysis of the corresponding image and retain the continuous image content at that time point. Reverse dark texture compensation is added to the continuous images that are paused during analysis to weaken the bright areas formed by refracted bright lines. The compensated continuous images are then reused for real-time recognition to form a recognition result with refraction interference correction.
[0015] The AI vision-based vending machine item recognition system includes a rotation sequence construction module, a refraction trajectory generation module, a contour stitching and reconstruction module, a risk timing determination module, and a dynamic compensation and control module. The rotation sequence construction module collects continuous images of the transparent packaging of the vending machine during the picking and placing process, and unifies the rotation changes in the continuous images to the same time rhythm to generate a rotation sequence of the transparent packaging; The refraction trajectory generation module tracks the movement trajectory of the refracted bright line in the continuous image during the transparent packaging rotation sequence, and generates a refraction interference trajectory draft based on the time position of the refracted bright line covering the edge of the transparent packaging. The contour stitching and reconstruction module extracts edge fragments that are not obscured by refracted bright lines from the images before and after the refraction interference trajectory record, and continuously stitches the edge fragments to reconstruct the transparent packaging contour, thereby generating a candidate set of transparent packaging contours. The risk timing determination module compares the real-time recognition results with the transparent packaging outline candidate set and generates a record of the time of misjudgment risk based on the outline contraction position and the outline breakage position. The dynamic compensation and control module adjusts the acquisition rhythm and lighting sequence of continuous images according to the time record of misjudgment risk. At the time position of misjudgment risk, the analysis operation is paused and reverse dark pattern compensation is added. The refraction interference of the real-time recognition result is corrected to output the final recognition result.
[0016] The technical effects and advantages provided by the present invention in the above technical solution are as follows: This invention constructs a transparent packaging rotation sequence within continuous images and records the movement trajectory of the refracted bright lines frame by frame, enabling a continuous temporal representation of the occlusion caused by the refracted bright lines on the real edges. The resulting refractive interference trajectory draft allows the true outline of the transparent packaging to be recovered during the refractive coverage. By extracting undisturbed edge fragments from images before and after coverage and continuously stitching them together, the true outline of the transparent packaging during dynamic handling can be re-presented, significantly improving outline integrity and providing a stable data foundation for subsequent recognition.
[0017] This invention records the time of misjudgment risk, enabling the real-time recognition process to proactively avoid the influence of refraction interference at critical moments. It adjusts the continuous image acquisition rhythm and supplementary lighting sequence at risk locations, pauses analysis on risky images, and adds reverse dark texture compensation. This weakens the bright bands formed by refracted bright lines during the risky period, restoring the true edges of transparent packaging that are obscured. The compensated image is then re-involved in recognition, ensuring that the final output recognition result remains stable in dynamic scenes involving transparent materials, unaffected by false edge interference caused by refraction, thus enhancing the accuracy and reliability of item recognition in vending machines. Attached Figure Description
[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.
[0019] Figure 1 This is a flowchart of the AI vision-based item recognition method for vending machines according to the present invention.
[0020] Figure 2 This is a schematic diagram of the module of the AI vision-based unmanned vending machine item recognition system of the present invention. Detailed Implementation
[0021] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that the description of this disclosure will be more complete and fully convey the concept of the exemplary embodiments to those skilled in the art.
[0022] This invention provides, for example Figure 1 The AI vision-based item recognition method for vending machines shown includes the following steps: Collect continuous footage of the transparent packaging in the vending machine during the picking and placing process, and unify the rotation changes in the continuous footage to the same time rhythm to generate a transparent packaging rotation sequence; In the implementation of the method for recognizing transparent packaging in vending machines, to ensure a coherent representation of the posture changes of the transparent packaging during the handling process, and to enable subsequent processing to accurately analyze the behavior of the transparent packaging in consecutive instants according to time sequence, the specific processing steps revolve around acquiring continuous images, organizing the image content, adjusting the rhythm of rotation changes, and uniformly arranging the continuous images on the timeline. The specific implementation steps are as follows: The system continuously captures footage before the user touches the transparent packaging of the vending machine. This captures the entire process, including the packaging at rest, the moment the user touches it, the rotational motion as the user pushes it, the posture changes caused by minor collisions with surrounding objects, the tilting changes due to uneven force applied by the user, and the slight swaying as the packaging regains balance. The continuous footage is generated at fixed time intervals, ensuring a complete record of the packaging's state at each moment. To prevent time gaps between frames, the system maintains continuity between each frame, ensuring the packaging's posture, position, and relative relationship with its environment remain continuously extended along the timeline. This process allows the entire process of the transparent packaging—from rest to rotation, and then to being moved or repositioned—to be fully recorded, enabling the continuous footage to reflect the true dynamic changes of the packaging throughout the entire handling process.
[0023] After obtaining the continuous footage, the footage is arranged sequentially according to the rotation and movement of the transparent packaging within the frames. This ensures the continuous footage accurately reflects the changes in the transparent packaging's posture during actual handling. During this arrangement, the rotation amplitude of the transparent packaging in different frames is continuously observed. Frames with large rotation amplitudes are arranged sequentially with those with small rotation amplitudes, presenting the footage according to the natural progression of the transparent packaging's changes. If the transparent packaging undergoes a significant change in posture due to a sudden change in rotation direction caused by the user's action, these frames are kept closely adjacent to the preceding and following frames to maintain a continuous flow of rotation within the sequence. Frames showing slow rotation of the transparent packaging are supplemented by inserting them sequentially along the timeline, allowing the overall sequence to present a gradually changing rotation posture. Through observation and sequential arrangement of rotation amplitudes, the resulting continuous footage accurately depicts the actual changes in the transparent packaging throughout the handling process, enabling subsequent steps to continue with a time-based processing rhythm based on this arrangement.
[0024] After organizing the sequence of consecutive frames, the rhythm of these frames is adjusted according to a unified time step, ensuring a consistent presentation rhythm across the time dimension. Because the transparent packaging may experience momentary acceleration, deceleration, brief stops of rotation, and sudden reversals of rotation due to external contact during handling, directly using the original frame time distribution would result in inconsistent intervals between frames, causing jumps in the packaging's posture changes on the timeline. Therefore, during rhythm adjustment, the organized consecutive frames are redistributed at fixed time intervals, maintaining a fixed distance between each frame and the preceding and following frames on the timeline. This rhythm adjustment ensures that the transparent packaging's rotational changes on the timeline present a continuous and uniform variation, allowing changes in its different postures to be presented with a fixed temporal rhythm. This process stabilizes the presentation sequence of the consecutive frames in the time dimension, enabling subsequent processing steps to perform motion tracking and posture analysis based on a unified rhythm.
[0025] After adjusting the timing, the continuous frames processed with unified timing are combined sequentially to form a transparent packaging rotation sequence. This rotation sequence incorporates all the posture changes of the transparent packaging during each stage of handling, presenting the entire process—from a stationary position, through rotation, swinging, tilting, returning to center, being pushed, stopping, and possibly being touched again—as a continuous sequence. Each frame in the transparent packaging rotation sequence is arranged according to a fixed timing rhythm, ensuring the rotational changes are continuous. This sequence not only records the posture changes of the transparent packaging throughout the entire movement but also presents the outline position, angle changes, lighting changes, hand occlusion changes, and surface reflection changes of the transparent packaging in consecutive instants. This allows subsequent steps to track the position of refracted bright lines and extract unobstructed edge segments in the time dimension based on this sequence. The formation of the transparent packaging rotation sequence provides a complete record of the dynamic changes of the transparent packaging, laying the temporal foundation for subsequent steps such as generating refraction interference trajectories, edge segment splicing, and outline reconstruction. It also enables subsequent steps to perform positioning processing on the timeline according to a unified rhythm.
[0026] Track the movement trajectory of the refracted bright line in the continuous image during the transparent packaging rotation sequence, and generate a refraction interference trajectory draft based on the time position of the refracted bright line covering the edge of the transparent packaging. After the transparent packaging rotation sequence is formed, the movement of the refracted bright line throughout the entire handling process is fully recorded, focusing on how the refracted bright line appears in the continuous images. Based on this, a refraction interference trajectory is generated that reflects the moment the refracted bright line obscures the edge of the transparent packaging. This allows subsequent steps to identify the interference status of the transparent packaging edge at different time points based on the trajectory. The specific implementation steps are as follows: In each frame of the rotating transparent packaging sequence, the area where the refracted bright line appears in the image is located, ensuring accurate recording of its position in each frame. The refracted bright line is produced when the rotating transparent packaging surface is illuminated by light. Its shape may appear as a horizontal stretch, vertical extension, arc refraction, short strip, local diffusion, or multiple continuous reflections in different frames. Therefore, when recording the position of the refracted bright line, its complete shape in the image is included in the recording scope. The recorded content includes the extension direction of the refracted bright line in consecutive frames, the bright area of the refracted bright line, the changes in the sharpness of the refracted bright line's edges, the offset position of the refracted bright line when sliding on the transparent packaging surface, and the instantaneous offset that may occur when the transparent packaging undergoes a slight oscillation. This ensures that the entire presentation behavior of the refracted bright line can be presented in a single frame. By recording the complete area of the refracted bright line in each frame, subsequent steps can use this recorded content to observe the movement trend of the refracted bright line over time.
[0027] The obtained single-frame records of the refracted bright lines are continuously correlated in chronological order, forming a complete movement trajectory for the refracted bright lines within the rotation sequence of the transparent packaging. During the correlation process, the position of the refracted bright lines in the previous frame is continuously compared with that in the next frame. This ensures that the direction, amplitude, and angle of movement of the refracted bright lines as they slide across the rotating transparent packaging surface, as well as the expansion changes that occur during rapid rotation, are reflected in the trajectory. As the angle of the transparent packaging changes during handling, the position of the refracted bright lines in each frame also changes. Therefore, in the trajectory correlation, every offset of the refracted bright lines at different time points is included in the continuous trajectory. This fully expresses possible trajectory bending, multiple segmentation, trajectory jumps, and oblique sliding on the transparent packaging surface during rotation. This continuous correlation method ensures the continuity of the movement path of the refracted bright lines throughout the entire frame sequence, providing a complete path basis for subsequently determining the relationship between the refracted bright lines and the edges of the transparent packaging.
[0028] After the movement path of the refracted bright line is formed, the trajectory of the refracted bright line in continuous frames is matched with the position of the transparent packaging edge frame by frame, ensuring that all moments when the refracted bright line covers the transparent packaging edge in the frame are completely recorded. The transparent packaging edge appears as a continuous outline in each frame. This outline shifts in angle, position, and length as the transparent packaging rotates and its posture changes. The refracted bright line may cover certain local areas of the transparent packaging edge at different times, causing the outline area of the transparent packaging to lose clarity in the frame. Therefore, in the matching process, the complete outline of the transparent packaging edge in each frame is matched with the area where the refracted bright line appears, allowing the specific segments of the refracted bright line covering the transparent packaging edge to be identified during the matching process. An independent record is generated for each coverage action, including the area of the transparent packaging edge obscured by the refracted bright line, the duration of the obscuration, the time and location of the obscuration, and the changes in the appearance of the edge outline after being covered by the refracted bright line during the obscuration process. This process allows the interaction between the refracted bright lines and the transparent packaging edge to be fully presented in both time and space dimensions throughout the entire process, providing a basis for the subsequent generation of the refractive interference trajectory draft.
[0029] After completing the frame-by-frame correspondence processing between the refracted bright line trajectory and the coverage position of the transparent packaging edge, the position changes, path changes, coverage behavior changes, and coverage duration of the refracted bright line throughout the entire transparent packaging rotation sequence are integrated in chronological order. This ensures that the refraction interference trajectory draft accurately presents the complete interference performance of the refracted bright line in continuous images. During the integration process, the presentation area of the refracted bright line in each frame, the movement path of the refracted bright line in continuous images, the specific location of the refracted bright line covering the transparent packaging edge, and the temporal location of the coverage behavior are all included in the trajectory draft. This allows the trajectory draft to clearly show the distribution of interference intensity, interference coverage area, and interference duration of the refracted bright line throughout the entire handling process. After the refraction interference trajectory draft is formed, subsequent steps can use the trajectory draft to determine which edge segments of the transparent packaging can be used for contour reconstruction at certain time periods, and which edge segments cannot be used due to being covered by the refracted bright line. This allows the entire processing to complete the analysis and recording of refraction interference on a continuous time axis, providing a complete interference basis for subsequent edge segment splicing and transparent packaging contour reconstruction.
[0030] Extract edge segments that are not obscured by refracted bright lines from the images before and after the coverage of the refracted interference trajectory record, and continuously stitch the edge segments to reconstruct the transparent packaging outline, thereby generating a candidate set of transparent packaging outlines; After clearly marking the locations where refracted bright lines covered the edge of the transparent packaging at different times in the refraction interference trajectory draft, in order to restore the true outline of the transparent packaging under the presence of refraction interference in continuous images, edge fragments were extracted, continuously stitched, and outline-forming processes were performed around the images before and after the coverage marked in the refraction interference trajectory draft. This allowed the true shape of the transparent packaging in continuous motion to be gradually reconstructed through these fragments. The specific implementation steps are as follows: Using each time point of the refracted bright line coverage behavior recorded in the refraction interference trajectory as the starting point for analysis, each frame before the coverage behavior occurs is considered the pre-coverage frame, and each frame after the coverage behavior ends is considered the post-coverage frame. The edge presentation of the transparent packaging in these frames is observed one by one. In the pre-coverage frame, since the refracted bright line has not yet formed an obstruction effect, the edge lines of the transparent packaging in the frame usually remain intact and continuous, presenting a natural contour shape composed of the corners, curves, sides, bottle shoulders, bottle bottoms, or box edges of the transparent packaging surface. In the post-coverage frame, since the refracted bright line has left the surface of the transparent packaging, the edges of the transparent packaging reappear, and may present new angles and shapes after the light is redistributed. By extracting edge lines from the images before and after the overlay frame by frame, the edge segments formed by the transparent packaging at different points in time are preserved in their original form. This allows each edge segment to fully reflect the posture changes of the transparent packaging at the corresponding point in time, including the contour shift caused by the transparent packaging rotating to the left, the outer contour extension caused by the transparent packaging rotating to the right, the edge height changes caused by the transparent packaging during slight tilting, and the overall contour movement caused by the transparent packaging being pushed away by the user. This ensures that each segment can represent the true shape of the transparent packaging at a specific moment.
[0031] Edge fragments extracted from the pre- and post-overlay images are arranged according to the chronological order of the transparent packaging's rotation sequence, ensuring that the arrangement of these edge fragments on the timeline reflects the posture changes of the transparent packaging during continuous movement. During this arrangement, the position of each edge fragment in the image, its extension direction, its relative orientation to other parts of the transparent packaging, and its offset in different images are sorted according to the transparent packaging's temporal position throughout the entire movement. This places edge fragments from earlier images at the beginning of the timeline and edge fragments from the post-overlay image at the end, clearly demonstrating the positional changes of the transparent packaging before, during, and after overlay through the arrangement of edge fragments. In this sorting process, all posture changes generated by the transparent packaging during rotation are incorporated into the edge fragment sorting criteria, ensuring that the edge fragments exhibit a continuous trend of change from the initial to the middle and final stages of rotation, providing a foundation for temporal consistency in subsequent splicing operations.
[0032] After chronologically organizing the edge segments, the sorted edge segments are continuously spliced together according to the spatial positions of the transparent packaging at different points in time, allowing the outer contour of the transparent packaging to reconstruct a complete line from multiple time segments. During the splicing process, the end positions of the edge segments in the previous image are spatially aligned with the starting positions of the edge segments in the subsequent image, allowing any missing parts between the two edge segments to be naturally filled in by the changes in the posture of the transparent packaging during movement. For example, when the transparent packaging rotates, causing the edge to gradually move from the left side to the center of the image, aligning the extension directions and curvature changes of multiple segments creates a continuous contour line between the edge segments; when the transparent packaging swings and partially expands or contracts, connecting the edge segments corresponding to these local changes creates a continuous trend in the overall shape; when the transparent packaging is gently pushed by the user and shifts as a whole, aligning the corresponding points of the edge segments before and after the shift creates a continuous shape in the contour segments due to the actual displacement. This splicing method gradually connects multiple segments into a complete outline, allowing the true outline of the transparent packaging to be recovered through edge information in adjacent images even when the refracted bright lines are obscured.
[0033] After continuously stitching together the edge segments, the formed complete outlines are summarized and organized according to the different postures of the transparent packaging throughout the entire handling process, forming a candidate set of transparent packaging outlines. When constructing the candidate set, each completed outline formed by stitching is treated as an independent outline instance, and its corresponding time position, the direction of outline line extension, the degree of closure of the outline lines, the offset of the outline shape, the change in outline curvature, and the stability change of the outline in continuous images are recorded. This ensures that each outline instance reflects the true form of the transparent packaging in a specific posture. As the transparent packaging continues to move during handling, the complete outline shapes generated at different time positions may exhibit differences in angular offset, shape extension, local curvature changes, and overall positional changes. By including all these outlines in the candidate set of transparent packaging outlines, the various possible shapes of the transparent packaging can be preserved through the candidate set, providing a basis for subsequent steps to comprehensively judge the true outline of the transparent packaging.
[0034] The real-time recognition results are compared using a candidate set of transparent packaging contours, and a time record of misjudgment risk is generated based on the contour contraction position and contour breakage position. After the candidate set of transparent packaging outlines has been established through the previous steps, in order to establish a correspondence between the outline changes formed during the real-time recognition process of handling transparent packaging and the complete outline shape in the candidate set, and to accurately pinpoint all possible time points that could lead to recognition failure during real-time recognition, the following processing steps are taken: frame-by-frame recording of the real-time recognition outline, correspondence processing between the real-time outline and candidate outlines, centralized organization of shrinkage and breakage phenomena in the real-time outline, and recording the position of these phenomena on the timeline. This ensures that the final misjudgment risk time record accurately reflects the real recognition risk of transparent packaging in dynamic scenarios. The specific implementation steps are as follows: The outline of the transparent packaging is recorded frame by frame during real-time recognition, with each frame's real-time outline lines recorded separately. During recording, the extension direction of the real-time outline, the curve direction of the outline lines, the turning changes of the outline edges, the positional shift of the outline in the frame, the area enclosed by the outline, and the changes in the outline across consecutive frames are all included in the recording. This ensures that the outline shape of the real-time recognition result fully reflects the actual posture changes of the transparent packaging during continuous handling. For example, when the transparent packaging rotates due to the user's hand, the real-time outline will shift along the direction of rotation in the frame; when the transparent packaging sways slightly at a moment, the outline will show subtle positional changes across consecutive frames; when the transparent packaging reflects light to varying degrees under different lighting conditions, the real-time outline will show a brief outline shift in local areas. By fully recording these specific changes, each frame of the real-time outline can participate in the next step of comparison processing.
[0035] The recorded real-time contours are processed according to different contour shapes in the candidate set of transparent packaging contours, enabling the real-time contours to match a specific contour shape in the candidate contours. This matching process compares the extension direction of the real-time contour with that of the candidate contours, the curvature changes of the real-time contour with those of the candidate contours, the enclosed area of the real-time contour with that of the candidate contours, and the positional shift of the real-time contour in the image with the positional trend of the candidate contour during the transparent packaging's movement. This ensures that every part of the real-time contour can find its corresponding part in the candidate contours. During the comparison process, if the real-time contour exhibits edge shortening, edge shifting, abrupt edge changes, edge disappearance, or edge breakage at a certain position, this difference is recorded as a special marker. For example, when the real-time contour shrinks in the area covered by the refracted bright line, comparison with the candidate contour reveals that the extension length of the real-time contour in a certain direction is inconsistent with the candidate contour; when the real-time contour breaks when the refracted bright line obscures the edge of the transparent packaging, comparison with the candidate contour reveals that a continuous line cannot be formed at the corresponding contour position. This frame-by-frame comparison allows for further analysis of the deviation between the real-time contour and the complete contour in the next step.
[0036] After comparing the real-time contour with the candidate contours, the contour contraction and breakage points that occurred during the comparison process are compiled and organized. Each contraction or breakage point is then mapped to a specific time point in the transparent packaging rotation sequence, forming a complete temporal correlation. During the contour contraction point organization, points where the real-time contour is shorter, oriented inwards, has a smaller enclosed area, or insufficient extension compared to the candidate contour in a given frame are recorded as contraction points. Similarly, points where the real-time contour is interrupted, disconnected, missing, or segmented compared to the candidate contour in a given frame are recorded as breakage points. The presence of these points during the transparent packaging's movement often indicates discontinuous changes in real-time recognition. Examples include rapid rotation of the transparent packaging causing momentary image blurring, reflection interference causing partial edge disappearance, and partial obstruction by the user's hand leading to a discontinuous contour. By correlating these points with time, each abnormal contour change can be accurately located on the timeline.
[0037] All identified contour contraction and breakage locations were arranged sequentially according to the rotation sequence of the transparent packaging. This created a continuous record of all potential recognition failure points during real-time identification of the transparent packaging, which was then saved as a record of misjudgment risks. During integration, the time and location of each contour contraction, each contour breakage, the transition from a complete to a contracted state, the transition from a continuous to a broken state, and the recovery from an abnormal state to a complete state were all included in the record. This ensured that the record of misjudgment risks fully reflected the identification difficulties of the transparent packaging throughout the entire handling process. In this way, the critical time periods when the real-time identification results were affected by refraction, occlusion, or posture changes during the movement were accurately represented, providing a continuous reference for adjusting the acquisition rhythm and implementing reverse dark pattern compensation in the next step.
[0038] The acquisition rhythm and lighting sequence of continuous images are adjusted according to the time record of misjudgment risk. The analysis operation is paused at the time of misjudgment risk and reverse dark pattern compensation is added. The refraction interference of the real-time recognition result is corrected to output the final recognition result. After the misjudgment risk time record has accurately identified the specific time locations of contour contraction and contour breakage of the transparent packaging in the continuous image through the preceding steps, in order to prevent the real-time recognition process from being affected by interference caused by refracted bright lines at these time locations, and to compensate for the image when interference occurs, a series of technical steps are carried out, including adjusting the acquisition rhythm of the continuous image, adjusting the supplementary lighting sequence, pausing the analysis operation, and adding reverse dark texture compensation. This ensures that the entire recognition process can maintain a stable recognition effect during the dynamic movement of the transparent packaging, and finally outputs the recognition result corrected for refraction interference. The specific implementation steps are as follows: Each time position marked in the misjudgment risk time record is used as a trigger point for adjusting the continuous image acquisition rhythm. During the continuous image acquisition process, the time position of each frame is judged in chronological order, so that the frames in the misjudgment risk time record can be identified as special time periods. After identifying these special time periods, the continuous image acquisition rhythm is adjusted from the original fixed time step to a more intensive time step, so that the frames in the risk time period can be captured at more detailed time intervals. During the adjustment process, the posture changes caused by the rapid rotation of transparent packaging, the brief spatial fluctuations of transparent packaging due to the instantaneous touch of the user's hand, the enhanced reflection behavior of transparent packaging under sudden changes in light conditions, and the rapid displacement behavior of transparent packaging when it is pushed inside the cargo channel are continuously captured, so that the image content that may cause misjudgment during the recognition process can be preserved in a more complete form. Through this rhythm adjustment method, the image information of the misjudgment risk time position is made more concentrated, creating a sufficient image foundation for subsequent lighting adjustment and compensation processing.
[0039] After adjusting the capture rhythm of continuous footage based on the recorded misjudgment risk times, the location of the misjudgment risk time was used as the trigger point for adjusting the supplementary lighting sequence. This caused the transparent packaging to receive a lighting sequence with a different direction than the original lighting during these time periods. During the adjustment of the supplementary lighting sequence, the direction of the light source was switched according to the spatial lighting characteristics of the risk time period. For example, the lighting from the left side of the image was switched to the right side, or the lighting from the top of the image was switched to the side, thus changing the incident angle of the light on the transparent packaging during the risk time period. After the change in lighting direction, the refracted bright lines on the surface of the transparent packaging may change from a straight line to an arc reflection, or shift from being concentrated in a fixed area of the transparent packaging to another area, changing the way the refracted bright lines cover the edges of the transparent packaging. Through this change, the reflection area formed by the refracted bright lines at the risk time is no longer fixed, thereby reducing the occlusion range of the transparent packaging edges caused by the refracted bright lines. Through the adjustment of the lighting sequence, the edge features of the image during the risk time period are more complete, allowing subsequent reverse shadow compensation to restore the outline of the transparent packaging based on more realistic lighting.
[0040] After adjusting the continuous image acquisition rhythm and the supplementary lighting sequence, the analysis operation for the risk time location is paused based on the recorded time position of the misjudgment risk time. This prevents the real-time recognition process from immediately performing contour analysis on the image content during the risk period. During the paused analysis, the image content of the risk period is completely saved as a continuous image buffer segment, allowing these images to be used as the original basis for subsequent compensation processing. During the pause, any enhancement of refracted bright lines, sudden disappearance of edges, amplification of localized high-brightness reflections, or blurred contours formed by the transparent packaging during the risk period will not be directly used as the basis for recognition by the real-time recognition process. By pausing, the real-time recognition process avoids making incorrect judgments during the risk period, maintaining a stable rhythm in the entire recognition process in terms of time logic. The image content after the pause provides the original information foundation for subsequent reverse dark texture compensation, allowing compensation to be performed on the complete image content.
[0041] After rhythm adjustment, fill light sequence adjustment, and resolution pause are completed, reverse dark freckle compensation is added to the image processing during the misjudged risk period. This restores the area where the refracted bright lines cover the edge of the transparent packaging to its normal brightness range. During reverse dark freckle compensation, the brightness of the high-brightness areas formed by the refracted bright lines is suppressed frame by frame, allowing the edges of the transparent packaging originally covered by the refracted bright lines to reappear in the image. As the edge loss areas caused by the refracted bright lines gradually recover, these recovered edge contents are continuously aligned with the contours before and after the risk period, forming a continuous edge trajectory. After reverse dark freckle compensation is completed, the compensated image is reintroduced into the real-time recognition process, ensuring that the final recognition result presents the true contour of the transparent packaging, rather than the false contour caused by the refracted bright lines. By having both the compensated and normal images participate in the final recognition, the recognition process can completely eliminate refraction interference, thus outputting a final recognition result corrected for refraction interference.
[0042] This invention constructs a transparent packaging rotation sequence within continuous images and records the movement trajectory of the refracted bright lines frame by frame, enabling a continuous temporal representation of the occlusion caused by the refracted bright lines on the real edges. The resulting refractive interference trajectory draft allows the true outline of the transparent packaging to be recovered during the refractive coverage. By extracting undisturbed edge fragments from images before and after coverage and continuously stitching them together, the true outline of the transparent packaging during dynamic handling can be re-presented, significantly improving outline integrity and providing a stable data foundation for subsequent recognition.
[0043] This invention records the time of misjudgment risk, enabling the real-time recognition process to proactively avoid the influence of refraction interference at critical moments. It adjusts the continuous image acquisition rhythm and supplementary lighting sequence at risk locations, pauses analysis on risky images, and adds reverse dark texture compensation. This weakens the bright bands formed by refracted bright lines during the risky period, restoring the true edges of transparent packaging that are obscured. The compensated image is then re-involved in recognition, ensuring that the final output recognition result remains stable in dynamic scenes involving transparent materials, unaffected by false edge interference caused by refraction, thus enhancing the accuracy and reliability of item recognition in vending machines.
[0044] This invention provides, for example Figure 2 The AI vision-based vending machine item recognition system shown includes a rotation sequence construction module, a refraction trajectory generation module, a contour stitching and reconstruction module, a risk timing determination module, and a dynamic compensation and control module. The rotation sequence construction module collects continuous images of the transparent packaging of the vending machine during the picking and placing process, and unifies the rotation changes in the continuous images to the same time rhythm to generate a rotation sequence of the transparent packaging; The refraction trajectory generation module tracks the movement trajectory of the refracted bright line in the continuous image during the transparent packaging rotation sequence, and generates a refraction interference trajectory draft based on the time position of the refracted bright line covering the edge of the transparent packaging. The contour stitching and reconstruction module extracts edge fragments that are not obscured by refracted bright lines from the images before and after the refraction interference trajectory record, and continuously stitches the edge fragments to reconstruct the transparent packaging contour, thereby generating a candidate set of transparent packaging contours. The risk timing determination module compares the real-time recognition results with the transparent packaging outline candidate set and generates a record of the time of misjudgment risk based on the outline contraction position and the outline breakage position. The dynamic compensation and control module adjusts the acquisition rhythm and lighting sequence of continuous images according to the time record of misjudgment risk. At the time position of misjudgment risk, the analysis operation is paused and reverse dark pattern compensation is added. The refraction interference of the real-time recognition result is corrected to output the final recognition result.
[0045] The AI vision-based vending machine item recognition method provided in this embodiment of the invention is implemented through the aforementioned AI vision-based vending machine item recognition system. For details of the specific methods and processes of the AI vision-based vending machine item recognition system, please refer to the embodiments of the aforementioned AI vision-based vending machine item recognition method, which will not be repeated here.
[0046] The foregoing has only described certain exemplary embodiments of the present invention by way of illustration. Undoubtedly, those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the foregoing drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.
Claims
1. A method for item recognition in unmanned vending machines based on AI vision, characterized in that, Includes the following steps: Collect continuous footage of the transparent packaging in the vending machine during the taking and putting process, and unify the rotation changes in the continuous footage to the same time rhythm to generate a transparent packaging rotation sequence; Track the movement trajectory of the refracted bright line in the continuous image during the transparent packaging rotation sequence, and generate a refraction interference trajectory draft based on the time position of the refracted bright line covering the edge of the transparent packaging. Extract edge segments that are not obscured by refracted bright lines from the images before and after the coverage of the refracted interference trajectory record, and continuously stitch the edge segments to reconstruct the transparent packaging outline, thereby generating a candidate set of transparent packaging outlines; The real-time recognition results are compared using a candidate set of transparent packaging contours, and a time record of misjudgment risk is generated based on the contour contraction position and contour breakage position. The acquisition rhythm and lighting sequence of continuous images are adjusted according to the time record of misjudgment risk. The analysis operation is paused at the time position of misjudgment risk and reverse dark pattern compensation is added. The refraction interference of the real-time recognition result is corrected to output the final recognition result.
2. The method for item recognition in an unmanned vending machine based on AI vision according to claim 1, characterized in that, The steps for generating the transparent packaging rotation sequence are as follows: The system captures continuous footage of transparent packaging during the handling and placement process. The continuous footage covers the entire process of the transparent packaging being stationary, in contact, rotating, tilting, swinging, and moving, and forms a continuous sequence of footage at fixed time intervals. The rotation and movement of transparent packaging in a continuous sequence of images are organized sequentially, and the continuous images are arranged according to the order of posture changes to form a sequence of images reflecting the process of taking and putting away transparent packaging. The timing rhythm of consecutive shots is adjusted according to the order of the shots, and the consecutive shots are redistributed according to a unified time step to form a shot arrangement with a consistent timing rhythm. The continuous images with adjusted timing are combined in chronological order to generate a transparent packaging rotation sequence, which is used to express the continuous posture changes of the transparent packaging during the handling process.
3. The method for item recognition in an unmanned vending machine based on AI vision according to claim 2, characterized in that, The steps for generating the refraction interference trajectory draft are as follows: Locate the area in which the refracted bright line appears in the continuous frame of the transparent packaging rotation sequence, and record the position and shape of the refracted bright line in each frame. Based on the records of refracted bright lines in each frame, the correlation processing is performed according to the time sequence of the transparent packaging rotation sequence to form the movement trajectory of the refracted bright lines in the continuous frames. The movement trajectory of the refracted bright line is correlated with the outline position of the transparent packaging edge in each frame, and the time position when the refracted bright line covers the transparent packaging edge is recorded. The movement trajectory of the refracted bright line is integrated with the time position of covering the edge of the transparent packaging to form a refracted interference trajectory draft, which is used to characterize the interference of the refracted bright line during the picking and placing process.
4. The method for item recognition in an unmanned vending machine based on AI vision according to claim 3, characterized in that, When recording the time and position of the refracted bright line covering the edge of the transparent packaging, the coverage area and duration of the refracted bright line in the corresponding image are recorded simultaneously, and written into the refraction interference trajectory file according to the time sequence of the transparent packaging rotation sequence, in order to distinguish the changes in the intensity of refraction interference at different time positions.
5. The method for item recognition in an unmanned vending machine based on AI vision according to claim 3, characterized in that, The steps for generating the candidate set of transparent packaging outlines are as follows: In the coverage time position marked by the refraction interference trajectory, distinguish between the continuous images before the refraction bright line coverage behavior occurs and the continuous images after the coverage behavior ends, and obtain the images before and after coverage. Extract the edge lines of the transparent packaging at the corresponding time positions in the images before and after the overlay, forming edge segments that reflect the changes in the posture of the transparent packaging. According to the time sequence of the transparent packaging rotation, the position, extension direction and posture changes of the edge segments in the picture are sorted out, and the edge segments are arranged according to the time sequence. Spatial alignment and continuous connection of edge segments arranged in chronological order are performed to reconstruct the outer contour of the transparent packaging, and the resulting outer contour is organized into a candidate set of transparent packaging contours.
6. The method for item recognition in an unmanned vending machine based on AI vision according to claim 5, characterized in that, In the process of spatial alignment and continuous connection of edge segments, the edge segments at adjacent time positions are correlated according to the orientation change direction of the transparent packaging in the rotation sequence, and continuous connection is made by combining the extension direction and curvature change relationship of the edge segments in the picture to form an outer contour that matches the picking and putting action of the transparent packaging. The candidate set of transparent packaging contours is defined to reflect the continuous shape change of the transparent packaging under different postures.
7. The method for item recognition in an unmanned vending machine based on AI vision according to claim 5, characterized in that, The steps for generating a time record of misjudgment risk are as follows: Record the outline shape of the transparent packaging in continuous images during real-time recognition, forming a real-time outline record corresponding to each frame; Based on the comparison between real-time contour records and transparent packaging contour candidate sets, the differences between real-time contours and candidate contours in terms of extension direction, curve direction and positional changes are analyzed. During the comparison process, the contour contraction and contour breakage locations that appear in the real-time contour are sorted out, and the contour contraction and contour breakage locations are associated with the time positions of the transparent packaging rotation sequence. The contour shrinkage and contour breakage positions are arranged according to the time sequence of the transparent packaging rotation sequence to form a time record of misjudgment risk that characterizes real-time risk identification.
8. The method for item recognition in an unmanned vending machine based on AI vision according to claim 7, characterized in that, When creating the misjudgment risk time record, the time intervals corresponding to the contour contraction position and the time intervals corresponding to the contour breakage position are marked respectively. The time distribution of different types of contour anomalies is distinguished in the misjudgment risk time record to indicate the risk source of contour changes during real-time identification.
9. The method for item recognition in an unmanned vending machine based on AI vision according to claim 7, characterized in that, Based on the time record of misjudgment risk, the acquisition rhythm and lighting sequence of continuous images were adjusted. At the corresponding time positions, the analysis operation was paused and reverse dark fringing compensation was added. The steps to correct refraction interference in the real-time recognition results are as follows: Based on the time record of misjudgment risk, the time position of continuous images is determined, and the acquisition rhythm of continuous images is adjusted at the corresponding time position to form a densely acquired sequence of continuous images. Based on the adjusted continuous image sequence, the order of supplementary lighting is changed at the time position corresponding to the time of misjudgment risk, and the light incident direction of the transparent packaging at the corresponding time position is adjusted. At the time points when the acquisition rhythm and fill light sequence are adjusted, pause the analysis of the corresponding image and retain the continuous image content at that time point. Reverse dark texture compensation is added to the continuous images that are paused during analysis to weaken the bright areas formed by refracted bright lines. The compensated continuous images are then reused for real-time recognition to form a recognition result with refraction interference correction.
10. An AI vision-based vending machine item recognition system, used to implement the AI vision-based vending machine item recognition method according to any one of claims 1-9, characterized in that, It includes a rotation sequence construction module, a refraction trajectory generation module, a contour stitching and reconstruction module, a risk timing determination module, and a dynamic compensation and control module. The rotation sequence construction module collects continuous images of the transparent packaging of the vending machine during the picking and placing process, and unifies the rotation changes in the continuous images to the same time rhythm to generate a rotation sequence of the transparent packaging; The refraction trajectory generation module tracks the movement trajectory of the refracted bright line in the continuous image during the transparent packaging rotation sequence, and generates a refraction interference trajectory draft based on the time position of the refracted bright line covering the edge of the transparent packaging. The contour stitching and reconstruction module extracts edge fragments that are not obscured by refracted bright lines from the images before and after the refraction interference trajectory record, and continuously stitches the edge fragments to reconstruct the transparent packaging contour, thereby generating a candidate set of transparent packaging contours. The risk timing determination module compares the real-time recognition results with the transparent packaging outline candidate set and generates a record of the time of misjudgment risk based on the outline contraction position and the outline breakage position. The dynamic compensation and control module adjusts the acquisition rhythm and lighting sequence of continuous images according to the time record of misjudgment risk. At the time position of misjudgment risk, the analysis operation is paused and reverse dark pattern compensation is added. The refraction interference of the real-time recognition result is corrected to output the final recognition result.