A machine vision-based cabinet article access identification method

Abstract

The application discloses a cabinet article access recognition method based on machine vision and relates to the technical field of intelligent cabinet article recognition, and comprises the following steps: acquiring a cabinet video and a cabinet door state, completing cabinet compartment registration and intercepting a current access event image sequence; extracting a foreground response and determining texture consistency, generating a short-time occlusion responsibility graph sequence; screening an occlusion recession frame to obtain a cabinet state image before and after access; combining a cabinet compartment calibration graph and a historical state snapshot to generate a compartment responsibility embedding feature; generating a directed change feature through a compartment occlusion responsibility enhanced SNUNet-CD network; according to the directed change feature and the historical snapshot, attributing an access type and closing a state, and outputting a recognition result and a current snapshot. The application utilizes machine vision and an occlusion responsibility enhanced change detection network, realizes cabinet article access recognition and state closure, and has the advantages of less occlusion misjudgment, accurate compartment attribution and stable state updating.

Description

Technical Field

[0001] This invention relates to the field of smart cabinet item recognition technology, and in particular to a machine vision-based method for identifying and retrieving items inside a cabinet. Background Technology

[0002] With the increasing application of smart cabinets, tool cabinets, consumable cabinets, and unmanned storage cabinets in scenarios such as warehouse management, medical consumable management, and office supplies management, the industry typically uses cameras to capture images inside the cabinets and determines whether items have been placed or removed through methods such as target detection, image comparison, fixed area recognition, or weight sensing. Existing vision solutions are mostly based on single-frame image recognition or differences between images before and after storage or retrieval, which can achieve basic recognition when the cabinet position is fixed, the appearance of the items is significantly different, and there are no obstructions during operation. However, in actual storage and retrieval processes, the opening of the cabinet door will cause changes in lighting, and the movement of hands, cuffs, and items will create short-term obstructions. Boundary misjudgments between adjacent cabinets are also prone to occur, making it difficult for ordinary image difference areas to directly correspond to the actual addition, loss, or displacement of items.

[0003] To alleviate the aforementioned problems, some existing technologies employ background modeling, foreground detection, local texture judgment, or depth change detection networks to improve recognition stability. For example, foreground segmentation can remove operational interference, or dual-temporal change detection networks can compare images before and after storage. However, these solutions typically treat occlusion areas as noise, failing to transform short-term occlusion sources into responsibility information for subsequent change recognition. Furthermore, ordinary change detection networks generally only output the changed area, lacking the participation of the cabinet space responsibility range and historical item state snapshots, making it difficult to distinguish between insertion, removal, displacement, and occlusion residue, and also difficult to continuously close the recognition result with the cabinet state of the next storage / retrieval recognition. Therefore, existing technologies still suffer from problems such as occlusion misjudgment, misassignment across cabinets, unclear storage / retrieval direction, and easy drift of state snapshots.

[0004] Therefore, how to provide a machine vision-based method for identifying and retrieving items inside a cabinet is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] One objective of this invention is to propose a machine vision-based method for identifying items stored and retrieved inside a cabinet. This invention utilizes machine vision and a responsibility-enhanced change detection network to achieve identification and state closure of items stored and retrieved inside the cabinet, and has the advantages of fewer occlusion errors, accurate compartment assignment, and stable state updates.

[0006] A method for identifying and retrieving items inside a cabinet based on machine vision according to an embodiment of the present invention includes the following steps: Acquire the video stream inside the cabinet and the opening and closing status of the cabinet door, perform cabinet grid coordinate registration on the video stream inside the cabinet, capture images based on the opening and closing status of the cabinet door, and generate the current access event image sequence; Perform SuBSENSE foreground response extraction and RLBSP local texture consistency determination on the current access event image sequence to generate a short-term occlusion responsibility map sequence; Based on the short-term occlusion responsibility map sequence, the current access event image sequence is subjected to occlusion fading filtering to generate the cabinet status image before access and the cabinet status image after access. Obtain the cabinet calibration map and historical item status snapshots; determine the cabinet space responsibility scope based on the cabinet calibration map; determine the historical occupancy status of the cabinet based on the historical item status snapshots; and generate cabinet responsibility embedding features. The state images of the cabinet before access, the state images of the cabinet after access, the short-term occlusion responsibility map sequence, and the grid occlusion responsibility embedding features are input into the grid occlusion responsibility enhanced SNUNet-CD network to perform responsibility constraint twin nested change recognition and generate a set of directed change features. Based on the set of directional change features and historical item state snapshots, the system performs access type attribution and state snapshot closure, generating the item access identification result and the current item state snapshot.

[0007] Optionally, the generation of the current access event image sequence specifically includes: The image frames of the video stream inside the cabinet are read according to the acquisition time sequence. The opening and closing status of the cabinet door is aligned to the image frames according to the same acquisition time sequence. The image frames of the cabinet door changing from closed to open and the image frames of the cabinet door changing from open to closed are identified to determine the image capture range of the current access event. Within the image capture range, identify the fixed boundaries of the cabinet and the intersection of the shelves, establish the mapping from the pixel position inside the cabinet to the coordinate position of the cabinet grid, perform cabinet grid coordinate registration on the image frames within the image capture range, and generate the registered event image frames. Arrange the registered event image frames according to the acquisition time sequence to generate the current access event image sequence.

[0008] Optionally, the generation of the short-term occlusion responsibility map sequence specifically includes: The earliest image frame in the current access event image sequence is used as the background initialization source to form the SuBSENSE background model; Based on the SuBSENSE background model, background matching is performed frame by frame on the current access event image sequence, and image positions that do not belong to the background are included in the operation foreground candidate region. RLBSP local texture consistency determination is performed on the candidate foreground region to eliminate foreground responses caused only by brightness jumps and generate local texture deviation regions. The foreground candidate region and the local texture deviation region are overlapped and filtered at the same image frame position, and the region that moves continuously along the direction of entry of the cabinet is retained to generate short-term occlusion candidate regions. A short-term occlusion responsibility map sequence is generated based on the image frame position of the short-term occlusion candidate region in the current access event image sequence.

[0009] Optionally, the generation of the cabinet state image before and after access specifically includes: The area of ​​the occlusion responsibility region is statistically analyzed frame by frame for the short-term occlusion responsibility map sequence. The image frame in which the occlusion responsibility region first appears and the image frame in which the occlusion responsibility region last disappears are identified to determine the filtering range before and after access. Calculate the pre-state stable residual of each candidate image frame within the pre-access screening range, select the candidate image frame with the smallest pre-state stable residual, and generate the pre-access cabinet state image. After accessing the database, calculate the post-state stable residual for each candidate image frame within the filtering range, select the candidate image frame with the smallest post-state stable residual, and generate the post-access state image of the cabinet.

[0010] Optionally, the generation of the grid responsibility embedding feature specifically includes: Obtain cabinet calibration diagrams and historical item status snapshots; Align the cabinet grid calibration diagram with the cabinet grid coordinate position, and generate the cabinet grid space responsibility area based on the outer boundary and internal placement area of ​​each cabinet grid. Based on the cabinet space responsibility scope, locate the historical item occupied area in the historical item status snapshot, and generate the cabinet historical occupancy status; Based on the scope of responsibility of the cabinet space, a cabinet location code is generated, and based on the historical occupancy status of the cabinet, a cabinet occupancy code is generated. Then, the codes are spliced ​​together according to the coordinate positions of the same cabinets to generate the cabinet responsibility embedding feature.

[0011] Optionally, the generation of the directed change feature set specifically includes: The state images of the front cabinet, the state images of the back cabinet, the short-term occlusion responsibility map sequence, and the grid responsibility embedding features are input into the grid occlusion responsibility enhanced SNUNet-CD network. The dual-temporal multi-scale features of the state images inside the cabinet before and after access are extracted by using a shared weighted twin coding path. The short-term occlusion responsibility map sequence and grid responsibility embedding features are converted into responsibility alignment features by responsibility embedding alignment path; By employing a nested decoding path for responsibility injection, hierarchical responsibility injection is performed on dual-temporal multi-scale features and responsibility-aligned features to generate responsibility fusion and change features. By processing the new and missing output headers in the directed change output path, the responsibility fusion change features are generated, and the new and missing change features are generated. By processing newly added variation features, missing variation features, and responsibility alignment features through the occlusion residual output head and shift output head in the directed variation output path, a set of directed variation features is generated.

[0012] Optionally, the grid occlusion responsibility-enhanced SNUNet-CD network includes a shared weighted twin coding path, a responsibility embedding alignment path, a responsibility injection nested decoding path, and a directed change output path.

[0013] Optionally, the shared weighted twin coding path includes a pre-access coding branch and a post-access coding branch. The pre-access coding branch receives the pre-access cabinet state image, and the post-access coding branch receives the post-access cabinet state image. The pre-access coding branch and the post-access coding branch adopt the same network structure and share convolution parameters. Both branches perform feature coding step by step in the order of convolution extraction, nonlinear activation, and downsampling.

[0014] Optionally, the generation of the cabinet item access identification result and the current item status snapshot specifically includes: Align the set of directed change features with the cabinet space responsibility scope and the historical item state snapshot to generate cabinet change alignment results; Based on the cabinet change alignment results, the newly added and missing change features are assigned to storage and retrieval types, and the insertion and retrieval statuses are generated. Based on the cabinet change alignment results, the displacement change characteristics are assigned to displacement status, and displacement status is generated. Based on the occlusion and residue characteristics, assign the occlusion and residue to the cabinet and retain the original historical occupancy status in the historical item status snapshot. Based on the placement status, retrieval status, shift status, and original historical occupancy status, perform a status snapshot closure to generate a current item status snapshot; Based on the insertion status, retrieval status, relocation status, and original historical occupancy status, the system generates the item storage and retrieval identification results in the cabinet, and uses the current item status snapshot as the historical item status snapshot for the next storage and retrieval identification.

[0015] The beneficial effects of this invention are: This invention captures the current access event image sequence by checking the cabinet door opening and closing status, and performs cabinet grid coordinate registration on the video stream inside the cabinet. This ensures that all image frames in the same access event are processed under the same cabinet grid coordinate position, avoiding the problem of difficulty in comparing before and after state images due to cabinet door opening and closing, shooting angle shift, or inconsistent cabinet grid boundaries. Furthermore, this invention uses SuBSENSE foreground response extraction and RLBSP local texture consistency determination to generate a short-term occlusion responsibility map sequence, and filters the cabinet state images before and after access based on the image frames where the occlusion responsibility area first appears and last disappears. This ensures that the dual-temporal images entering the change recognition network avoid the stages of hand, cuff, lighting jumps, and unstable item placement, reducing the impact of short-term operation interference on the access recognition results from the source.

[0016] This invention transforms cabinet calibration maps and historical item status snapshots into cabinet responsibility embedding features, enabling each cabinet coordinate location to simultaneously carry the cabinet's spatial responsibility range and historical occupancy status. This feature, along with a short-term occupancy responsibility map sequence, is input into a cabinet occupancy responsibility-enhanced SNUNet-CD network. Compared to ordinary change detection methods that only compare differences between images before and after access, this invention fuses cabinet ownership, historical occupancy, and occupancy source information within the network through responsibility embedding alignment paths and responsibility injection nested decoding paths. This further distinguishes change responses into newly added change features, missing change features, displacement change features, and occupancy residue features, thereby improving the accuracy of distinguishing insertion, retrieval, and displacement behaviors and reducing the problems of misattribution of adjacent cabinets and misjudging occupancy residue as genuine access changes.

[0017] This invention also performs access type attribution and state snapshot closure based on a set of directed change features and historical item state snapshots. The original historical occupancy states corresponding to the placement, retrieval, displacement, and occlusion residues are incorporated into the current item state snapshot generation process, and the current item state snapshot is used as the historical item state snapshot for the next access identification. Therefore, the identification result is no longer a single image judgment result, but can maintain state inheritance and closed-loop updates in continuous access events, reducing the risk of item state drift in long-term operation and improving the stability, traceability, and practical usability of item access identification in scenarios such as smart cabinets, tool cabinets, and consumable cabinets. Attached Figure Description

[0018] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of a machine vision-based method for identifying and retrieving items inside a cabinet, as proposed in this invention. Figure 2This is a flowchart of the occlusion responsibility generation process for a machine vision-based cabinet item storage and retrieval identification method proposed in this invention. Figure 3 This is a flowchart illustrating the directed change recognition process of a machine vision-based cabinet item retrieval and identification method proposed in this invention. Detailed Implementation

[0019] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0020] refer to Figures 1-3 A machine vision-based method for recognizing and retrieving items inside a cabinet includes the following steps: Acquire the video stream inside the cabinet and the opening and closing status of the cabinet door, perform cabinet grid coordinate registration on the video stream inside the cabinet, capture images based on the opening and closing status of the cabinet door, and generate the current access event image sequence; Perform SuBSENSE foreground response extraction and RLBSP local texture consistency determination on the current access event image sequence to generate a short-term occlusion responsibility map sequence; Based on the short-term occlusion responsibility map sequence, the current access event image sequence is subjected to occlusion fading filtering to generate the cabinet status image before access and the cabinet status image after access. Obtain the cabinet calibration map and historical item status snapshots; determine the cabinet space responsibility scope based on the cabinet calibration map; determine the historical occupancy status of the cabinet based on the historical item status snapshots; and generate cabinet responsibility embedding features. The state images of the cabinet before access, the state images of the cabinet after access, the short-term occlusion responsibility map sequence, and the grid occlusion responsibility embedding features are input into the grid occlusion responsibility enhanced SNUNet-CD network to perform responsibility constraint twin nested change recognition and generate a set of directed change features. Based on the set of directional change features and historical item state snapshots, the system performs access type attribution and state snapshot closure, generating the item access identification result and the current item state snapshot.

[0021] In this embodiment, the generation of the current access event image sequence specifically includes: The image frames of the video stream inside the cabinet are read according to the acquisition time sequence. The opening and closing status of the cabinet door is aligned to the image frames according to the same acquisition time sequence. The image frames of the cabinet door changing from closed to open and the image frames of the cabinet door changing from open to closed are identified to determine the image capture range of the current access event. Within the image capture range, identify the fixed boundaries of the cabinet and the intersection of the shelves, establish the mapping from the pixel position inside the cabinet to the coordinate position of the cabinet grid, perform cabinet grid coordinate registration on the image frames within the image capture range, and generate the registered event image frames. Arrange the registered event image frames according to the acquisition time sequence to generate the current access event image sequence.

[0022] In this embodiment, the generation of the short-term occlusion responsibility map sequence specifically includes: The earliest image frame in the current access event image sequence is used as the background initialization source to form the SuBSENSE background model; The SuBSENSE background model includes a color background component generated according to the cabinet coordinate position, an RLBSP texture background component, a color similarity boundary, and a texture similarity boundary; The color background component is the pixel color of the earliest acquired image frame in the current access event image sequence at the same cabinet coordinate position. The color similarity boundary is generated by the color fluctuation between the cabinet coordinate position and its surrounding neighborhood. The pixel color at the cabinet coordinate position is calculated with the pixel colors of each neighboring position in the surrounding neighborhood. The obtained Euclidean color distances are sorted by numerical value and the median color distance is taken. Then, the absolute difference between each Euclidean color distance and the median color distance is calculated. The obtained absolute differences are sorted by numerical value and the median color distance difference is taken. The median color distance is added to three times the median color distance difference to generate the color similarity boundary. The RLBSP texture background component is the RLBSP texture description of the earliest acquired image frame in the current access event image sequence at the same cabinet coordinate position. The RLBSP texture description takes the pixel corresponding to the cabinet coordinate position as the center pixel, selects the neighborhood sampling points around the center pixel, calculates the difference between the gray value of the center pixel and the gray value of each neighborhood sampling point, sorts the obtained gray value difference by numerical size and takes the median gray value difference, then calculates the absolute difference between each gray value difference and the median gray value difference, sorts the obtained absolute difference by numerical size and takes the median gray value difference, adds the median gray value difference to twice the median gray value difference to generate the gray value stability amount at the cabinet coordinate position, marks the neighborhood sampling points corresponding to the gray value difference that does not exceed the gray value stability amount as similar positions, and marks the neighborhood sampling points corresponding to the gray value difference that exceeds the gray value stability amount as dissimilar positions, and arranges the similar positions and dissimilar positions according to the fixed order of the neighborhood sampling points to generate the RLBSP texture description. The texture similarity boundary is generated by the binary difference between the RLBSP texture description at the same cabinet coordinate position and the RLBSP texture description at each neighboring position in the surrounding neighborhood. RLBSP texture descriptions are generated for each neighboring position. The RLBSP texture descriptions at each neighboring position are aligned with the RLBSP texture description at the same cabinet coordinate position according to the fixed order of the neighborhood sampling points. The number of inconsistencies between similar and dissimilar positions is counted to obtain the number of differences per neighboring position. The differences are sorted by numerical value and the median difference is taken. The absolute difference between each number of differences per position and the median difference is calculated. The absolute differences are sorted by numerical value and the median difference of texture differences is taken. The median difference is added to twice the median difference of texture differences and rounded up to generate the texture similarity boundary. Based on the SuBSENSE background model, background matching is performed frame by frame on the current access event image sequence, and image positions that do not belong to the background are included in the operation foreground candidate region. The background matching includes color background matching and RLBSP texture background matching. Color background matching compares the pixel color at the same cabinet coordinate position in the current image frame with the color background component. Image positions where the color difference does not exceed the color similarity limit are considered color backgrounds. RLBSP texture background matching compares the RLBSP texture description at the same cabinet coordinate position in the current image frame with the RLBSP texture background component bit by bit. Image positions where the number of bit-by-bit differences does not exceed the texture similarity limit are considered texture backgrounds. When the same image position does not belong to both color background and texture background simultaneously, the image position is included in the operation foreground candidate region. RLBSP local texture consistency determination is performed on the candidate foreground region to eliminate foreground responses caused only by brightness jumps and generate local texture deviation regions. The RLBSP local texture consistency determination generates a current RLBSP texture description for the image position in the operation foreground candidate region, and compares the current RLBSP texture description with the RLBSP texture background component at the same cabinet coordinate position bit by bit. Image positions whose bit-by-bit differences exceed the texture similarity limit are classified into the local texture deviation region, and image positions whose bit-by-bit differences do not exceed the texture similarity limit and only have color differences are removed from the operation foreground candidate region. The foreground candidate region and the local texture deviation region are overlapped and filtered at the same image frame position, and the region that moves continuously along the direction of entry of the cabinet is retained to generate short-term occlusion candidate regions. The cabinet entrance direction is the direction from the cabinet door opening to the depth inside the cabinet in the cabinet coordinate position; when the same image position belongs to both the operation foreground candidate area and the local texture deviation area, and the image position changes continuously along the cabinet entrance direction in adjacent image frames, the image position is classified into the short-term occlusion candidate area; the image position that appears only in a single image frame or does not change continuously along the cabinet entrance direction is not classified into the short-term occlusion candidate area. Generate a short-term occlusion responsibility map sequence according to the image frame position of the short-term occlusion candidate region in the current access event image sequence; Each short-term occlusion responsibility map in the short-term occlusion responsibility map sequence corresponds to an image frame in the current access event image sequence. Short-term occlusion candidate regions are marked as occlusion responsibility regions, and image positions not included in short-term occlusion candidate regions are marked as non-occlusion responsibility regions.

[0023] In this embodiment, the generation of the cabinet status image before and after access specifically includes: The area of ​​the occlusion responsibility region is statistically analyzed frame by frame for the short-term occlusion responsibility map sequence. The image frame in which the occlusion responsibility region first appears and the image frame in which the occlusion responsibility region last disappears are identified to determine the filtering range before and after access. The area of ​​the occlusion responsibility region is the number of image positions marked as occlusion responsibility regions in the short-term occlusion responsibility map; the image frame in which the occlusion responsibility region first appears is the image frame in which the area of ​​the occlusion responsibility region changes from zero to non-zero; the image frame in which the occlusion responsibility region last disappears is the image frame in which the area of ​​the occlusion responsibility region changes from non-zero to zero; image frames whose acquisition time is earlier than the first appearance of the occlusion responsibility region are included in the pre-access filtering range, and image frames whose acquisition time is later than the last disappearance of the occlusion responsibility region are included in the post-access filtering range; Calculate the pre-state stable residual of each candidate image frame within the pre-access screening range, select the candidate image frame with the smallest pre-state stable residual, and generate the pre-access cabinet state image. The pre-state stable residual is generated by accumulating the non-occluded image position differences between the candidate image frame and its adjacent next image frame. The non-occluded image position differences are generated by normalizing and fusing the color difference component and the RLBSP texture difference component. The color difference component is generated by dividing the Euclidean color distance of the pixel color at the same cabinet coordinate position in the candidate image frame and its adjacent next image frame by the maximum Euclidean color distance of all non-occluded image positions within the same candidate image frame. When the maximum Euclidean color distance is zero, the color difference component is recorded as zero. The RLBSP texture difference component is generated by aligning the RLBSP texture description at the same cabinet coordinate position in the candidate image frame and its adjacent next image frame bit by bit according to the fixed order of the neighborhood sampling points, counting the number of inconsistencies between similar and dissimilar bits, and dividing by the total number of neighborhood sampling points. The arithmetic mean of the color difference component and the RLBSP texture difference component is taken to generate the non-occluded image position difference at the cabinet coordinate position. Calculate the post-state stable residual for each candidate image frame within the filtering range after access, select the candidate image frame with the smallest post-state stable residual, and generate the state image inside the cabinet after access. The post-state stable residual is generated by accumulating the positional differences of the candidate image frame and its adjacent next image frame.

[0024] In this embodiment, the generation of the grid responsibility embedding feature specifically includes: Obtain cabinet calibration diagrams and historical item status snapshots; The cabinet calibration diagram is generated from the cabinet fixed boundary, shelf boundary and cabinet number in the cabinet empty calibration image. It is used to represent the spatial distribution of each cabinet in the cabinet coordinate position. The historical item status snapshot is obtained by closing the previous status snapshot. It is generated from the empty cabinet confirmation status when the access identification is first performed. It is used to represent the cabinet confirmation status before the current access event occurs. It includes the cabinet where the confirmed item is located, the historical item occupied area and the historical occupancy status. The historical occupancy status includes occupied status, partially occupied status and empty status. Align the cabinet grid calibration diagram with the cabinet grid coordinate position, and generate the cabinet grid space responsibility area based on the outer boundary and internal placement area of ​​each cabinet grid. The cabinet space responsibility range is generated in units of cabinet coordinate position. Cabinet coordinate positions located within the outer boundary of the same cabinet are assigned to the internal placement area of ​​that cabinet. Cabinet coordinate positions located between the outer boundaries of adjacent cabinets are assigned to the cabinet with the smaller distance according to the shortest distance to the internal placement area of ​​the adjacent cabinet. This generates a cabinet space responsibility range that covers the cabinet state image before and after access. Based on the cabinet space responsibility scope, locate the historical item occupied area in the historical item status snapshot, and generate the cabinet historical occupancy status; The historical occupancy status of the cabinet is generated by the position of the historical item occupancy area within the cabinet space responsibility area. The historical item occupancy area in the historical item status snapshot is overlapped with the cabinet space responsibility area. When the historical item occupancy area falls into a cabinet space responsibility area, the corresponding cabinet is recorded as occupied. When the historical item occupancy area spans multiple cabinet space responsibility areas, the local occupancy status of the corresponding cabinet is generated according to the overlapping area in each cabinet. Cabinets that do not overlap with the historical item occupancy area are recorded as vacant. Based on the scope of responsibility of the cabinet space, a cabinet location code is generated, and based on the historical occupancy status of the cabinet, a cabinet occupancy code is generated. Then, the codes are spliced ​​together according to the coordinate positions of the same cabinets to generate the cabinet responsibility embedding feature. The grid responsibility embedding feature has the same grid coordinate position as the grid state image before and after access. At each grid coordinate position, the grid responsibility embedding feature simultaneously carries the grid space responsibility range and the grid historical occupancy status to which the position belongs.

[0025] In this embodiment, the generation of the directed change feature set specifically includes: The state images of the front cabinet, the state images of the back cabinet, the short-term occlusion responsibility map sequence, and the grid responsibility embedding features are input into the grid occlusion responsibility enhanced SNUNet-CD network. The dual-temporal multi-scale features of the state images inside the cabinet before and after access are extracted by using a shared weighted twin coding path. The short-term occlusion responsibility map sequence and grid responsibility embedding features are converted into responsibility alignment features by responsibility embedding alignment path; The responsibility embedding alignment path includes an occlusion responsibility coding branch and a grid responsibility alignment branch. The occlusion responsibility coding branch counts the number of image frames marked as occlusion responsibility areas at the same grid coordinate position in the short-term occlusion responsibility map sequence, and divides the number of image frames by the total number of image frames in the current access event image sequence to generate the occlusion responsibility intensity. The occlusion responsibility coding branch numbers the first image frame in the current access event image sequence as 1 according to the acquisition time sequence, increments the number of subsequent image frames sequentially, divides the number of the image frame first marked as an occlusion responsibility area at the same grid coordinate position by the total number of image frames to generate the occlusion start time sequence value, and records the last image frame marked as an occlusion responsibility area at the same grid coordinate position. The image frame number of the occlusion responsibility area is divided by the total number of image frames to generate the occlusion end sequence value. When the same cabinet coordinate position is not marked as an occlusion responsibility area in the short-term occlusion responsibility map sequence, the occlusion responsibility intensity, occlusion start sequence value, and occlusion end sequence value of that cabinet coordinate position are all recorded as zero. The occlusion start sequence value and occlusion end sequence value are arranged according to the cabinet coordinate position to generate the occlusion timing code. The grid responsibility alignment branch splices the grid responsibility embedding feature, occlusion responsibility intensity, and occlusion timing code according to the same cabinet coordinate position, and performs regional average downsampling according to the spatial dimensions of each level of the dual-temporal multi-scale feature to generate responsibility alignment features at each scale. By employing a nested decoding path for responsibility injection, hierarchical responsibility injection is performed on dual-temporal multi-scale features and responsibility-aligned features to generate responsibility fusion and change features. The responsibility injection nested decoding path includes multiple nested decoding nodes, each corresponding to a scale level in the dual-temporal multi-scale features. In each nested decoding node, the pre-access multi-scale features, post-access multi-scale features, and responsibility alignment features at the same scale level are concatenated according to the channel direction, and convolutional fusion is performed on the concatenated features to generate the responsibility fusion node features at the corresponding scale level. The responsibility fusion node features at the previous scale level are upsampled and input into the nested decoding node at the next scale level, and are further concatenated and fused with the pre-access multi-scale features, post-access multi-scale features, and responsibility alignment features at the next scale level until a responsibility fusion change feature with the same cabinet coordinate position as the pre-access cabinet state image and the post-access cabinet state image is generated. By processing the new and missing output headers in the directed change output path, the responsibility fusion change features are generated, and the new and missing change features are generated. The directed change output path performs convolution mapping on the responsibility fusion change features to generate pre-access occupancy response and post-access occupancy response. The new output header subtracts the pre-access occupancy response from the post-access occupancy response to obtain a new response difference, retaining response positions greater than zero in the new response difference to generate a new change feature, representing a stable increase in occupancy in the cabinet's state image after access compared to the pre-access state image. The missing output header subtracts the post-access occupancy response from the pre-access occupancy response to obtain a missing response difference, retaining response positions greater than zero in the missing response difference to generate a missing change feature, representing a stable occupancy change where the cabinet's state image disappears before access compared to the post-access state image. Response positions with a new response difference not greater than zero are not included in the new change feature, and response positions with a missing response difference not greater than zero are not included in the missing change feature. By processing newly added change features, missing change features, and responsibility alignment features through the occlusion residual output head and shifted output head in the directed change output path, a set of directed change features is generated; The occlusion residual output head aligns the newly added and missing change features with the occlusion responsibility intensity at the same cabinet coordinate position. When the occlusion responsibility intensity at the cabinet coordinate position where the newly added or missing change feature is located is greater than zero, and the cabinet coordinate position is marked as an occlusion responsibility area in the short-term occlusion responsibility map corresponding to the occlusion start time value to the occlusion end time value, the corresponding change response is separated from the newly added or missing change feature and assigned to the occlusion residual feature. Within the cabinet space responsibility range carried by the grid responsibility embedding feature, the shift output head calculates the appearance similarity for the multi-scale features after access corresponding to the newly added change features that are not classified as occlusion residual features and the multi-scale features before access corresponding to the missing change features that are not classified as occlusion residual features. For the same newly added change feature, the missing change feature with the highest appearance similarity is used as the candidate pairing for the newly added side. For the same missing change feature, the newly added change feature with the highest appearance similarity is used as the candidate pairing for the missing side. When the candidate pairing for the newly added side and the candidate pairing for the missing side are consistent with each other and belong to different cabinet space responsibility ranges, the corresponding newly added change feature and missing change feature are combined into a shift change feature. The newly added change feature and missing change feature that have been combined into a shift change feature are no longer output as separate newly added change features or missing change features. The newly added change features that are not classified as shift change features and occlusion residual features, the missing change features that are not classified as shift change features and occlusion residual features, and the shift change features and occlusion residual features are combined into a directed change feature set. The appearance similarity is generated by dividing the inner product between the multi-scale feature after access corresponding to the newly added change feature and the multi-scale feature before access corresponding to the missing change feature by the product of the Euclidean length of the multi-scale feature after access corresponding to the newly added change feature and the Euclidean length of the multi-scale feature before access corresponding to the missing change feature. When any Euclidean length is zero, the corresponding appearance similarity is recorded as zero.

[0026] In this embodiment, the grid occlusion responsibility-enhanced SNUNet-CD network includes a shared weighted twin coding path, a responsibility embedding alignment path, a responsibility injection nested decoding path, and a directed change output path; The shared-weighted twin coding path is located in the feature extraction stage of the algorithm link. It receives the state images of the cabinet before and after access, and extracts dual-temporal multi-scale features at the same scale level in a unified feature representation space. The responsibility embedding and alignment path is located in the responsibility information construction stage of the algorithm link. It receives short-term occlusion responsibility map sequences and grid responsibility embedding features, and maps the responsibility information to a spatial scale consistent with the dual-temporal multi-scale features to generate responsibility alignment features. The responsibility injection nested decoding path is located in the change fusion stage of the algorithm link. It receives dual-temporal multi-scale features and responsibility alignment features, and generates responsibility fusion change features through multi-level feature fusion. The directed change output path is located in the change decision stage of the algorithm link. It receives responsibility fusion change features and outputs new change features, missing change features, shifted change features, and occlusion residual features according to the change direction and responsibility attributes. The grid occlusion responsibility-enhanced SNUNet-CD network is a structural improvement on the dual-temporal twin coding and nested decoding of the SNUNet-CD network. Ordinary SNUNet-CD networks typically only receive the images before and after the change, and output the changed region through twin coding and nested decoding paths. The grid occlusion responsibility-enhanced SNUNet-CD network further sets up a responsibility embedding alignment path and a directed change output path, using short-term occlusion responsibility map sequences and grid responsibility embedding features as internal network inputs to participate in the change recognition process. The short-term occlusion responsibility map sequence generates occlusion responsibility strength and occlusion temporal coding through the occlusion responsibility coding branch. The grid responsibility embedding features are concatenated with the occlusion responsibility strength and occlusion temporal coding at the same grid coordinate position through the grid responsibility alignment branch and downsampled to form a dual-temporal multi-scale... The responsibility alignment features are consistent with the scale of the feature; the responsibility injection nested decoding path simultaneously integrates pre-access multi-scale features, post-access multi-scale features, and responsibility alignment features in each level of nested decoding node, so that the change area is jointly restricted by the cabinet space responsibility range, the cabinet historical occupancy status, and the source of short-term occlusion during the decoding process, instead of performing grid matching after outputting the change map; through the directed change output path, the network no longer outputs a single change or no change result, but outputs newly added change features, missing change features, displacement change features, and occlusion residual features, distinguishing between insertion, removal, displacement, and unstable occlusion changes formed by hands, cuffs, shadows, or reflections, reducing the problems of misattribution of adjacent cabinets, misjudgment of short-term occlusion, and the inability of ordinary change detection to express the access direction, and improving the consistency between the cabinet item access recognition result and the current item state snapshot; The training data for the SNUNet-CD network with enhanced grid occlusion responsibility is constructed from multiple grid access events. Each training sample includes a grid state image before access, a grid state image after access, a short-term occlusion responsibility map sequence, grid responsibility embedding features, and a snapshot of historical item states. Each training sample is labeled with newly added change regions, missing change regions, shifted change regions, occlusion residual regions, and the corresponding access type. Newly added change regions are identified by the stable occupied areas formed in the grid state image after the item is placed; missing change regions are identified by the historical occupied areas in the grid state image before the item is removed; shifted change regions are identified by pairing the removal and insertion areas of the same item; and occlusion residual regions are identified by the change responses that appear in the short-term occlusion responsibility map sequence but do not form a stable access direction. During training, the grid state images before and after access, the short-term occlusion responsibility map sequence, and the grid responsibility embedding features are input into the network. The newly added change features, missing change features, shifted change features, and occlusion residual features are then calculated and correlated with their corresponding values. The network employs three loss mechanisms: regional loss, directional loss, and responsibility consistency loss. Regional loss is calculated by adding cross-entropy loss and Dice loss to constrain pixel-level consistency of newly added, missing, and occluded residual regions. Directional loss is calculated using access type cross-entropy loss to constrain the category assignment of insertion, removal, displacement, and occlusion residuals. Displacement matching loss is calculated by subtracting the appearance similarity between the features of the removed and inserted regions to improve the consistency of features before and after displacement of the same item. Responsibility consistency loss is calculated based on whether the changed features fall within the responsibility range of the cabinet space and whether they are consistent with the short-term occlusion responsibility map sequence marker position, to suppress cross-cabinet misassignment and misjudgment of occlusion residuals. The total loss is obtained by summing the regional loss, directional loss, displacement matching loss, and responsibility consistency loss, and the network parameters are updated through backpropagation. The network is considered converged when the total loss of the validation set decreases by less than one-thousandth for ten consecutive training rounds, and the average F1 score corresponding to newly added, missing, displacement, and occlusion residual features increases by less than one-thousandth for ten consecutive training rounds.

[0027] In this embodiment, the shared weighted twin coding path includes a pre-access coding branch and a post-access coding branch. The pre-access coding branch receives the state image inside the cabinet before access and the post-access coding branch receives the state image inside the cabinet after access. The pre-access coding branch and the post-access coding branch adopt the same network structure and share convolution parameters. Both of them perform feature coding step by step in the order of convolution extraction, nonlinear activation and downsampling. In the first-level coding layer, convolution operations are performed on the input image to extract cabinet edges, cabinet outlines, and item appearance texture features, generating the first-level feature map. In subsequent coding layers, the previous-level feature map is downsampled and the number of feature channels is increased to expand the receptive field and extract higher-level spatial semantic information. Each coding layer outputs a pre-access feature map and a post-access feature map at the corresponding scale. The pre-access and post-access feature maps at the same scale level are preserved according to their correspondence, forming a multi-level feature set from high resolution to low resolution. The high-resolution level represents cabinet boundaries, item outlines, and local texture information, while the low-resolution level represents the overall shape of the item, spatial occupancy, and cabinet semantic information. The pre-access feature maps at each scale level constitute pre-access multi-scale features, and the post-access feature maps at each scale level constitute post-access multi-scale features. The pre-access multi-scale features and post-access multi-scale features together constitute dual-temporal multi-scale features.

[0028] In this embodiment, the generation of the cabinet item access identification result and the current item status snapshot specifically includes: Align the set of directed change features with the cabinet space responsibility scope and the historical item state snapshot to generate cabinet change alignment results; The cabinet change alignment result is generated by the position of newly added change features, missing change features, shifted change features, and occlusion residual features within the cabinet space responsibility area; the cabinet coordinate position of each change feature is calculated to coincide with the cabinet space responsibility area to determine the cabinet to which the corresponding change feature belongs; the cabinet to which the corresponding change feature belongs is matched with the cabinet where the confirmed item is located, the historical item occupied area, and the historical occupancy status in the historical item status snapshot, so that each change feature corresponds to the cabinet confirmation status before the current access event occurs; Based on the cabinet change alignment results, the newly added and missing change features are assigned to storage and retrieval types, and the insertion and retrieval statuses are generated. The placement state is generated by matching the newly added change feature with the empty or partially occupied state in the historical item status snapshot. When the cabinet to which the newly added change feature belongs is empty in the historical item status snapshot, the corresponding newly added change feature is assigned to the placement state. When the cabinet to which the newly added change feature belongs is partially occupied in the historical item status snapshot and the newly added change feature does not overlap with the historical item's occupied area, the corresponding newly added change feature is assigned to the placement state. The removal state is generated by matching the missing change feature with the occupied or partially occupied state in the historical item status snapshot. When the missing change feature overlaps with the historical item's occupied area, the corresponding missing change feature is assigned to the removal state. Based on the cabinet change alignment results, the displacement change characteristics are assigned to displacement status, and displacement status is generated. The shift state is generated by the missing side change response and the new side change response in the shift change features. The cabinet to which the missing side change response belongs is the shifted cabinet, the cabinet to which the new side change response belongs is the shifted cabinet, the confirmed item that overlaps with the missing side change response in the historical item status snapshot is the shifted item, and the historical item occupied area of ​​the shifted item is transferred from the shifted cabinet to the area where the new side change response is located to generate the shift state. Based on the occlusion and residue characteristics, assign the occlusion and residue to the cabinet and retain the original historical occupancy status in the historical item status snapshot. The attribution of the occlusion residue is generated by matching the occlusion residue feature with the historical item state snapshot. When the cabinet to which the occlusion residue feature belongs is occupied or partially occupied in the historical item state snapshot, the historical item occupied area and historical occupied state of the corresponding cabinet are retained. When the cabinet to which the occlusion residue feature belongs is vacant in the historical item state snapshot, the vacant state of the corresponding cabinet is retained. The occlusion residue feature does not trigger the placement state, removal state, or displacement state. Based on the placement status, retrieval status, shift status, and original historical occupancy status, perform a status snapshot closure to generate a current item status snapshot; The state snapshot closure includes: writing the area where the newly changed feature corresponding to the placement state is located into the current item state snapshot; deleting the historical item-occupied area corresponding to the removal state from the current item state snapshot; updating the moved item corresponding to the displacement state from the removed cabinet to the moved-in cabinet; and maintaining the cabinet to which the obscuring residual feature belongs as the original historical occupied state in the historical item state snapshot. After completing the above updates, the historical occupied state is updated according to the occupancy status of each cabinet in the current item state snapshot, so that the cabinet with a complete item-occupied area is recorded as occupied, the cabinet with only cross-cabinet item-occupied part is recorded as partially occupied, and the cabinet without an item-occupied area is recorded as empty. Based on the insertion status, retrieval status, relocation status, and original historical occupancy status, the system generates the item storage and retrieval identification results in the cabinet, and uses the current item status snapshot as the historical item status snapshot for the next storage and retrieval identification.

[0029] Example 1: To verify the feasibility of this invention in practice, it was applied to a scenario of identifying the storage and retrieval of items in a cabinet within the consumables room of a tertiary hospital's operating room. This scenario involved six intelligent consumables cabinets, each containing 48 compartments. These cabinets stored consumables such as suture devices, hemostatic materials, catheters, and puncture kits, all of similar appearance and with highly reflective packaging. A fixed camera was installed on the top of each cabinet, and the cabinet door's open / closed status was output via a magnetic door switch. In actual use, nurses frequently insert their hands and cuffs into the cabinets when retrieving consumables. Some consumable packaging has a silver reflective film, and there are instances where the edges of items cross the boundaries of adjacent compartments. Conventional image difference methods before and after storage and retrieval can easily misinterpret hand obstruction, sudden changes in lighting, or changes in the boundaries of adjacent compartments as actual retrieval and placement, leading to inconsistencies between the cabinet's internal state and the actual storage state.

[0030] In this scenario, video streaming inside the cabinet is captured when the cabinet door changes from closed to open, and capture stops when the door changes back to closed, forming the current access event image sequence. Each image frame completes cabinet grid coordinate registration based on the fixed boundaries of the cabinet and the intersection of the shelves, ensuring that images from different times fall into the same grid coordinate position. The earliest captured image frame in the current access event image sequence is used to form the SuBSENSE background model, generating a color background component, an RLBSP texture background component, a color similarity boundary, and a texture similarity boundary at each grid coordinate position. Subsequently, background matching is performed frame by frame in the current access event image sequence, classifying image positions that do not belong to the background into the operation foreground candidate region. Then, the foreground response caused only by brightness jumps is eliminated by the RLBSP local texture consistency judgment, while the region that moves continuously along the cabinet entrance direction is retained, generating a short-term occlusion responsibility map sequence. The short-term occlusion responsibility map sequence is used to identify the first appearance and last disappearance of the occlusion responsibility area in the image frame. Within the corresponding range, the image frame with the smallest stable residual amount in the previous state and the smallest stable residual amount in the subsequent state is selected to obtain the state image inside the cabinet before access and the state image inside the cabinet after access, avoiding the use of the operator's hand, unstable consumables and reflection at the moment of opening the door as the change recognition input.

[0031] After generating the dual-temporal state images, the cabinet fixed boundaries, shelf boundaries, and cabinet numbers in the cabinet empty calibration image are used to generate the cabinet calibration map. The historical item state snapshot is provided by the closure result of the previous state snapshot. The cabinet space responsibility range is determined based on the cabinet calibration map, and the historical occupancy status of the cabinet is generated based on the historical item occupied area and historical occupancy status. Then, the cabinet position code and cabinet occupancy code are concatenated according to the same cabinet coordinate position to generate the cabinet responsibility embedding feature. The cabinet internal state image before access, the cabinet internal state image after access, the short-term occlusion responsibility map sequence, and the cabinet responsibility embedding feature are jointly input into the cabinet occlusion responsibility enhanced SNUNet-CD network. The network first extracts multi-scale features before and after access through a shared weighted Siamese coding path. Then, it converts the short-term occlusion responsibility map sequence into occlusion responsibility intensity and occlusion temporal code through a responsibility embedding alignment path, and concatenates and downsamples it with the cabinet responsibility embedding feature to form a responsibility alignment feature. The responsibility-injection nested decoding path fuses pre-access multi-scale features, post-access multi-scale features, and responsibility alignment features at each level of nested decoding node. Finally, the directed change output path generates new change features, missing change features, shift change features, and occlusion residual features. Based on the matching results of these directed change features with historical item state snapshots, placement state, retrieval state, and shift state are generated, while the cabinet corresponding to the occlusion residual features maintains its original historical occupancy state, thus completing the closure of the current item state snapshot.

[0032] To train the SNUNet-CD network with enhanced compartment occlusion responsibility, data on compartment access events were collected at the same location during the same period, resulting in 5280 training samples and 1320 validation samples. Each training sample included a compartment state image before and after access, a short-term occlusion responsibility map sequence, compartment responsibility embedding features, and a snapshot of the historical item state. Human reviewers labeled each sample with newly added change areas, missing change areas, shifted change areas, residual occlusion areas, and corresponding access types based on actual barcode scanning records, final inventory results, and video playback. Newly added change areas were identified from stable occupied areas formed in the compartment state image after the consumables were placed; missing change areas were identified from historical item occupied areas in the compartment state image before the consumables were removed; shifted change areas were paired with the removal and insertion areas of the same consumable; and residual occlusion areas were identified from change responses that appeared in the short-term occlusion responsibility map sequence but did not form a stable access direction. During training, the region loss is obtained by adding the cross-entropy loss and the Dice loss, used to constrain the pixel-level consistency of newly added, missing, and occluded residual regions; the orientation loss is obtained by the access type cross-entropy loss, used to constrain the category assignment of insertion, retrieval, shifting, and occluded residuals; the shift matching loss is obtained by subtracting the appearance similarity between the features of the shifted-out region and the features of the shifted-in region, used to enhance the consistency of features before and after the shifting of the same consumable; the responsibility consistency loss is calculated based on whether the changed features fall within the responsibility range of the cabinet space and the consistency with the position of the short-term occlusion responsibility map sequence marker, used to suppress cross-cabinet misassignment and misjudgment of occluded residuals. The total loss is obtained by summing the region loss, orientation loss, shift matching loss, and responsibility consistency loss, and the network parameters are updated through backpropagation. Training is considered converged when the total loss of the validation set decreases by less than one-thousandth for ten consecutive rounds of training, and the average F1 score corresponding to the newly added changed features, missing changed features, shifted changed features, and occluded residual features increases by less than one-thousandth for ten consecutive rounds of training.

[0033] After 30 days of continuous operation in the operating room supplies room of the hospital, 1860 real cabinet access events were selected for verification, including 742 insertion events, 861 retrieval events, 157 cabinet displacement events, and 100 events with occlusion interference but no actual access. Using the ordinary frame difference method before and after access, the access type recognition accuracy was 86.8%, the misclassification rate of adjacent cabinets was 7.6%, the occlusion misjudgment rate was 9.3%, and the consistency rate between the current item status snapshot and the manual inventory results was 88.5%. Using a common SNUNet-CD network to directly compare the cabinet status images before and after access, the access type recognition accuracy was 91.2%, the misclassification rate of adjacent cabinets was 5.1%, the occlusion misjudgment rate was 6.4%, and the consistency rate of the current item status snapshot was 92.7%. After adopting this invention, the access type recognition accuracy reached 96.5%, the misclassification rate of adjacent cabinets decreased to 2.2%, the occlusion misjudgment rate decreased to 2.7%, and the consistency rate of the current item status snapshot reached 97.1%. The above results show that the present invention can stably distinguish between insertion, removal, displacement and obstruction residue in actual cabinet environments with hand obstruction, packaging reflection, interference from adjacent cabinet boundaries and consumable displacement, thereby improving the accuracy of cabinet item storage and retrieval identification results and reducing state drift during continuous operation.

[0034] Table 1. Comparison of Stability of Item Retrieval and Identification in Cabinets

[0035] The above experiment used 1860 real cabinet storage and retrieval events as the verification objects, including 742 placement events, 861 retrieval events, 157 cabinet displacement events, and 100 events with occlusion interference but no real storage or retrieval. Table 1 shows that the frame difference method combined with HSV color histogram matching can achieve certain recognition results in simple placement and retrieval scenarios, with an accuracy rate of 86.8% for storage and retrieval type recognition. However, its displacement recognition F1 score is only 70.6%, and its occlusion residue recognition F1 score is only 68.9%, indicating that relying solely on pixel differences and color distribution comparisons is insufficient to distinguish between real changes in objects and short-term changes caused by hands, cuffs, or reflections. When the edge of an object crosses a cabinet compartment or an operator occludes a local area within the cabinet, this method easily misidentifies the occluded area as a newly added or missing area. Therefore, the misclassification rate of adjacent cabinet compartments reaches 7.6%, the occlusion misjudgment rate reaches 9.3%, and the current item status snapshot consistency rate is only 88.5%.

[0036] Compared to the frame difference method, the YOLOv8n object detection and DeepSORT trajectory matching method has stronger object localization capabilities, improving the accuracy of access type recognition to 89.6% and the placement recognition F1 to 90.8%, indicating that the object bounding box and trajectory-based recognition method can effectively capture the process of items entering or leaving the cabinet. However, this method still has shortcomings in recognizing consumables with similar shapes, reflective packaging, or partial occlusion, especially in the scenario of moving items within the cabinet. In such cases, items simply move from one cabinet to another without necessarily forming a clear entry or exit trajectory, resulting in a displacement recognition F1 of only 76.4%. Furthermore, this method relies on post-processing matching between the object detection bounding box and the cabinet area; when the detection bounding box shifts near the boundary of adjacent cabinets, it still results in a 6.3% misclassification rate of adjacent cabinets.

[0037] The SNUNet-CD dual-temporal change detection method directly compares the cabinet state images before and after access, enabling it to identify cabinet state changes more stably than target detection trajectory methods. It achieves an access type recognition accuracy of 91.2%, a displacement recognition F1 score of 81.6%, and a current item state snapshot consistency rate of 92.7%. However, this method primarily outputs the change area, lacking the participation of short-term occlusion responsibility map sequences and grid responsibility embedding features. It cannot distinguish between historical grid occupancy states, true stable changes, and operational occlusion residues within the network. Therefore, when hand occlusion, cuff occlusion, or reflective residues appear near the images before and after access, it easily outputs unstable changes as true changes. Its occlusion residue recognition F1 score is 79.2%, and the occlusion misclassification rate is 6.4%, indicating that relying solely on dual-temporal change detection is insufficient to meet the stability requirements of continuous state closure within the cabinet.

[0038] This invention achieves an access type recognition accuracy of 96.5% under the same verification data, which is 5.3 percentage points higher than the SNUNet-CD dual-temporal change detection method; the shift recognition F1 score reaches 90.4%, which is 8.8 percentage points higher than the SNUNet-CD dual-temporal change detection method; and the occlusion residue recognition F1 score reaches 90.7%, which is 11.5 percentage points higher than the SNUNet-CD dual-temporal change detection method. These improvements are mainly attributed to the synergistic effect between the short-term occlusion responsibility map sequence, the grid responsibility embedding features, and the grid occlusion responsibility-enhanced SNUNet-CD network. The short-term occlusion responsibility map sequence uses SuBSENSE foreground response extraction and RLBSP local texture consistency judgment to express the responsibility of operation occlusion, so that short-term changes such as hands, cuffs, and reflections are no longer directly included in the judgment of real access changes. The grid responsibility embedding feature introduces the grid space responsibility range and the grid historical occupancy status into the network, so that the change response can be restricted by the grid ownership and historical status during the decoding process. The directed change output path further distinguishes the change results into new change features, missing change features, displacement change features, and occlusion residue features, so that insertion, retrieval, displacement, and occlusion residue can be processed separately at the feature level, rather than being manually classified after outputting a single change area.

[0039] From the perspective of error rate metrics, this invention reduces the misclassification rate of adjacent cabinets to 2.2%, which is 2.9 percentage points lower than the SNUNet-CD dual-temporal change detection method and 4.1 percentage points lower than the YOLOv8n object detection and DeepSORT trajectory matching method. The occlusion misjudgment rate is reduced to 2.7%, which is 3.7 percentage points lower than the SNUNet-CD dual-temporal change detection method and 6.6 percentage points lower than the frame difference method and HSV color histogram matching. These results indicate that this invention not only improves the overall recognition accuracy but also achieves more significant improvements in addressing the occlusion misjudgment and cross-cabinet misclassification problems that are most prone to state drift in cabinet access scenarios. Although the average processing time per event for this invention is 0.39 seconds, which is higher than other comparative methods, this processing time still meets the application requirements for state confirmation after cabinet door opening and closing, and in return, achieves a higher consistency rate of the current item state snapshot. The consistency rate of the current item status snapshot was increased from 92.7% for the SNUNet-CD dual-phase change detection method to 97.1%, indicating that the present invention can maintain the status of items in the cabinet more stably in continuous access events by using access type attribution and status snapshot closure, thereby reducing the cumulative spread of misidentification in subsequent events.

[0040] In summary, this invention, when faced with issues such as packaging reflections, hand occlusion, cuff interference, adjacent compartment boundary interference, and item displacement within the cabinet, can mitigate the impact of unstable occlusion changes through short-term occlusion responsibility map sequences. It improves the accuracy of compartment assignment based on change responses by embedding compartment responsibility features and outputs a set of directed change features with access direction meaning through a compartment occlusion responsibility-enhanced SNUNet-CD network. Ultimately, it achieves stable differentiation of insertion, retrieval, displacement, and occlusion residue. Compared to frame difference methods and HSV color histogram matching, YOLOv8n object detection and DeepSORT trajectory matching, and SNUNet-CD dual-temporal change detection methods, this invention demonstrates superior performance in terms of recognition accuracy, displacement recognition capability, occlusion residue recognition capability, stability of adjacent compartment assignment, and consistency of current item state snapshots, effectively reducing the risk of state drift of items within the cabinet during long-term use.

[0041] The above are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for recognizing and retrieving items inside a cabinet based on machine vision, characterized in that, Includes the following steps: Acquire the video stream inside the cabinet and the opening and closing status of the cabinet door, perform cabinet grid coordinate registration on the video stream inside the cabinet, capture images based on the opening and closing status of the cabinet door, and generate the current access event image sequence; Perform SuBSENSE foreground response extraction and RLBSP local texture consistency determination on the current access event image sequence to generate a short-term occlusion responsibility map sequence; Based on the short-term occlusion responsibility map sequence, the current access event image sequence is subjected to occlusion fading filtering to generate the cabinet status image before access and the cabinet status image after access. Obtain the cabinet calibration map and historical item status snapshots; determine the cabinet space responsibility scope based on the cabinet calibration map; determine the historical occupancy status of the cabinet based on the historical item status snapshots; and generate cabinet responsibility embedding features. The state images of the cabinet before access, the state images of the cabinet after access, the short-term occlusion responsibility map sequence, and the grid occlusion responsibility embedding features are input into the grid occlusion responsibility enhanced SNUNet-CD network to perform responsibility constraint twin nested change recognition and generate a set of directed change features. Based on the set of directional change features and historical item state snapshots, the system performs access type attribution and state snapshot closure, generating the item access identification result and the current item state snapshot.

2. The method for identifying and retrieving items inside a cabinet based on machine vision according to claim 1, characterized in that, The generation of the current access event image sequence specifically includes: The image frames of the video stream inside the cabinet are read according to the acquisition time sequence. The opening and closing status of the cabinet door is aligned to the image frames according to the same acquisition time sequence. The image frames of the cabinet door changing from closed to open and the image frames of the cabinet door changing from open to closed are identified to determine the image capture range of the current access event. Within the image capture range, identify the fixed boundaries of the cabinet and the intersection of the shelves, establish the mapping from the pixel position inside the cabinet to the coordinate position of the cabinet grid, perform cabinet grid coordinate registration on the image frames within the image capture range, and generate the registered event image frames. Arrange the registered event image frames according to the acquisition time sequence to generate the current access event image sequence.

3. The method for identifying and retrieving items inside a cabinet based on machine vision according to claim 1, characterized in that, The generation of the short-term occlusion responsibility map sequence specifically includes: The earliest image frame in the current access event image sequence is used as the background initialization source to form the SuBSENSE background model; Based on the SuBSENSE background model, background matching is performed frame by frame on the current access event image sequence, and image positions that do not belong to the background are included in the operation foreground candidate region; RLBSP local texture consistency determination is performed on the candidate foreground region to eliminate foreground responses caused only by brightness jumps and generate local texture deviation regions. The foreground candidate region and the local texture deviation region are overlapped and filtered at the same image frame position, and the region that moves continuously along the direction of entry of the cabinet is retained to generate short-term occlusion candidate regions. A short-term occlusion responsibility map sequence is generated based on the image frame position of the short-term occlusion candidate region in the current access event image sequence.

4. The method for identifying and retrieving items inside a cabinet based on machine vision according to claim 1, characterized in that, The generation of the cabinet status image before and after access specifically includes: The area of ​​the occlusion responsibility region is statistically analyzed frame by frame for the short-term occlusion responsibility map sequence. The image frame in which the occlusion responsibility region first appears and the image frame in which the occlusion responsibility region last disappears are identified to determine the filtering range before and after access. Calculate the pre-state stable residual of each candidate image frame within the pre-access screening range, select the candidate image frame with the smallest pre-state stable residual, and generate the pre-access cabinet state image. After accessing the database, calculate the post-state stable residual for each candidate image frame within the filtering range, select the candidate image frame with the smallest post-state stable residual, and generate the post-access state image of the cabinet.

5. The method for identifying and retrieving items inside a cabinet based on machine vision according to claim 1, characterized in that, The generation of the grid responsibility embedding feature specifically includes: Obtain cabinet calibration diagrams and historical item status snapshots; Align the cabinet grid calibration diagram with the cabinet grid coordinate position, and generate the cabinet grid space responsibility area based on the outer boundary and internal placement area of ​​each cabinet grid. Based on the cabinet space responsibility scope, locate the historical item occupied area in the historical item status snapshot, and generate the cabinet historical occupancy status; Based on the scope of responsibility of the cabinet space, a cabinet location code is generated, and based on the historical occupancy status of the cabinet, a cabinet occupancy code is generated. Then, the codes are spliced ​​together according to the coordinate positions of the same cabinets to generate the cabinet responsibility embedding feature.

6. The method for identifying and retrieving items inside a cabinet based on machine vision according to claim 1, characterized in that, The generation of the directed change feature set specifically includes: The state images of the front cabinet, the state images of the back cabinet, the short-term occlusion responsibility map sequence, and the grid responsibility embedding features are input into the grid occlusion responsibility enhanced SNUNet-CD network. The dual-temporal multi-scale features of the state images inside the cabinet before and after access are extracted by using a shared weighted twin coding path. The short-term occlusion responsibility map sequence and grid responsibility embedding features are converted into responsibility alignment features by responsibility embedding alignment path; By employing a nested decoding path for responsibility injection, hierarchical responsibility injection is performed on dual-temporal multi-scale features and responsibility-aligned features to generate responsibility fusion and change features. By processing the new and missing output headers in the directed change output path, the responsibility fusion change features are generated, and the new and missing change features are generated. By processing newly added variation features, missing variation features, and responsibility alignment features through the occlusion residual output head and shift output head in the directed variation output path, a set of directed variation features is generated.

7. A method for identifying and retrieving items inside a cabinet based on machine vision according to claim 6, characterized in that, The grid-masking responsibility-enhanced SNUNet-CD network includes a shared-weight twin coding path, a responsibility embedding alignment path, a responsibility-injected nested decoding path, and a directed change output path.

8. A method for identifying and retrieving items inside a cabinet based on machine vision according to claim 6, characterized in that, The shared weighted twin coding path includes a pre-access coding branch and a post-access coding branch. The pre-access coding branch receives the state image inside the cabinet before access, and the post-access coding branch receives the state image inside the cabinet after access. The pre-access coding branch and the post-access coding branch adopt the same network structure and share convolution parameters. Both branches perform feature coding step by step in the order of convolution extraction, nonlinear activation, and downsampling.

9. A method for identifying and retrieving items inside a cabinet based on machine vision according to claim 1, characterized in that, The generation of the cabinet item access identification result and the current item status snapshot specifically includes: Align the set of directed change features with the cabinet space responsibility scope and the historical item state snapshot to generate cabinet change alignment results; Based on the cabinet change alignment results, the newly added and missing change features are assigned to storage and retrieval types, and the insertion and retrieval statuses are generated. Based on the cabinet change alignment results, the displacement change characteristics are assigned to displacement status, and displacement status is generated. Based on the occlusion and residue characteristics, assign the occlusion and residue to the cabinet and retain the original historical occupancy status in the historical item status snapshot. Based on the placement status, retrieval status, shift status, and original historical occupancy status, perform a status snapshot closure to generate a current item status snapshot; Based on the insertion status, retrieval status, relocation status, and original historical occupancy status, the system generates the item storage and retrieval identification results in the cabinet, and uses the current item status snapshot as the historical item status snapshot for the next storage and retrieval identification.

Images