Item surveillance system and method - Patents.com

Abstract

The order confirmation system includes a video sensor configured to capture video footage of a monitored area located in proximity to the receiving / shipping portal. The processing unit performs event analysis on the captured video footage to detect entities, detect arrival of goods from a third-party supplier from a door opening event, identify the third-party supplier, perform a check-in process for a delivery person associated therewith, detect entry and exit of goods through the receiving / shipping portal, and verify that the detected delivered products match data regarding products to be delivered by the third-party supplier. The database stores at least a dataset of face images / logos for face / brand detection and a dataset of product images for product identification. The database records the results of the order confirmation process and the check-out of the delivery person at the end of the delivery.

Description

[Technical field]

[0001] FIELD OF THE DISCLOSURE The present disclosure relates generally to systems and methods for monitoring, and more particularly, to systems and methods for monitoring incoming / outgoing goods at a receiving / shipping portal. [Background technology]

[0002] Environments, such as retail environments with warehouses and the like, may facilitate the entry and exit of people and goods in addition to the storage / retrieval of goods. Often, the high traffic of people and goods passing through these environments may pose challenges in tracking individual movements of people and goods into such environments to prevent, for example, entry of unauthorized persons, theft of goods, correct receipt / dispatch of goods, etc. Traditionally, these efforts may have been performed manually by deploying security personnel. However, when such traffic of people and goods is high, such manual efforts by security personnel are tedious and inconsistent. Thus, there is a need for a more robust system and method that can perform surveillance of people and goods entering and exiting such environments without requiring manual intervention. Summary of the Invention

[0003] In one aspect of the disclosure, an order confirmation system is provided. The order confirmation system comprises a plurality of video sensors adapted to capture video footage of a monitored area located within an order receiving or shipping area of ​​a receiving / shipping portal. The order confirmation system further comprises a processing unit configured to perform event analysis on the captured video footage, detect entities in the video footage captured by the video sensors, detect arrivals from third-party suppliers from door opening events in the captured video footage, identify the third-party suppliers, perform a check-in process for delivery personnel from the third-party suppliers, detect goods entering and leaving through the receiving / shipping portal, and verify that the detected delivery products match data regarding products to be delivered by the third-party suppliers. The order confirmation system also comprises a database communicatively coupled to the processing unit. The database is configured to store at least a dataset of face images / logos for use in face / brand detection and a dataset of product images for use in product identification. The database is also configured to record results of the order confirmation process at the end of delivery and the check-out of the delivery personnel for future retrieval upon request to the processing unit.

[0004] In another aspect of the disclosure, a method for performing video surveillance is provided. The method includes using a plurality of video sensors to capture video footage of a monitored area located within an order receiving or shipping area of ​​a receiving / shipping portal. The method further includes using a processing unit to perform event analysis on the captured video footage. The method further includes using the processing unit to detect an entity in the video footage captured by the video sensors. The method further includes using the processing unit to detect an arrival of a package from a third-party supplier from a door opening event in the captured video footage. The method further includes using the processing unit to identify the third-party supplier and perform a check-in process for a delivery person from the third-party supplier. The method further includes using the processing unit to detect the entry and exit of goods through the receiving / shipping portal and verify that the detected delivery product matches data regarding the product to be delivered by the third-party supplier. The method further includes using a database to store at least a dataset of face images / logos for use in face / brand detection and a dataset of product images for use in product identification. The method further includes using a database of the delivery person's check-out at the end of the order confirmation process and delivery to record the results. The method further includes retrieving the record by the processing unit from the database in response to a request to the processing unit.

[0005] In yet another aspect of the present disclosure, the embodiments disclosed herein are also directed to a non-transitory computer-readable medium having stored thereon computer-executable instructions that, when executed by a processing unit, cause the processing unit to perform the methods disclosed herein.

[0006] The present disclosure presents a system and method for monitoring of incoming / outgoing goods at a receiving / shipping portal. The disclosure is described with reference to a retail environment. However, those skilled in the art will appreciate that the disclosure is not limited to use in a retail environment. Instead, the disclosure is applicable to any environment in which goods pass through a shipping portal as they leave a first order processing facility, and then pass through a receiving portal as they enter a second order receiving facility, which is the goods' required destination. Thus, the objective of the disclosed system is to detect discrepancies between the planned inventory records of an incoming / outgoing order and the actual contents of the corresponding incoming / pre-shipped order.

[0007] Additionally, the system addresses the problem of detecting inaccurate or incomplete orders and similarly inaccurate or incomplete assembled orders prior to dispatch. In this manner, merchandise traffic at both ends, i.e., the shipping and receiving portals of the distribution system, can be characterized and controlled to improve both the accuracy of the distribution system and the inventory control process at both the order processing and order receiving facilities.

[0008] Incoming merchandise orders typically pass through an entry door of an order receiving facility prior to receipt and acceptance of the order by staff at the order receiving facility. Similarly, outgoing merchandise from an order fulfillment facility typically passes through an exit door of an order fulfillment facility prior to delivery to a required destination. For simplicity, the entry door of an order receiving facility will hereafter be referred to as a receiving portal. Similarly, the exit door of an order fulfillment facility will hereafter be referred to as a shipping portal. The present disclosure addresses the problem of loss prevention at receiving / shipping portals. In particular, the present disclosure enables automated confirmation of inbound / outgoing merchandise orders at a receiving / shipping portal.

[0009] In practice, an order receiving facility often includes multiple receiving portals. Similarly, an order fulfillment facility often includes multiple shipping portals. In fact, a given facility may undertake both order receiving and order fulfillment, in which case the facility may include a first multiple receiving portal and a second multiple shipping portal. The time required to manually review each order and the high volume of incoming and / or outgoing order traffic typically occurring at an order receiving facility / order fulfillment facility makes manual monitoring of incoming and outgoing orders very difficult. The challenge is amplified when multiple deliveries occur simultaneously within the limited space of the order receiving or order fulfillment area of ​​a receiving / shipping portal.

[0010] Accordingly, the present disclosure discloses a system and method, respectively, for automated monitoring of incoming and outgoing traffic from either or both of the receiving and shipping portals of an order receiving facility and an order fulfillment facility. For brevity, the system and method of the present disclosure are hereinafter referred to as the order confirmation system and the order confirmation method, respectively.

[0011] It will be understood that features of the present disclosure can be combined in various combinations without departing from the scope of the present disclosure, which is defined by the appended claims. [Brief description of the drawings]

[0012] The above summary, as well as the following detailed description of exemplary embodiments, will be better understood when read in conjunction with the accompanying drawings. For the purpose of illustrating the disclosure, exemplary structures of the disclosure are shown in the drawings. The disclosure is not limited, however, to the particular methods and instrumentalities disclosed herein. Moreover, those skilled in the art will appreciate that the drawings are not to scale. Whenever possible, similar elements are designated by the same numerals. [Figure 1] FIG. 1 is a perspective view of an exemplary environment in which an order confirmation system may execute, according to one embodiment of the present disclosure. [Diagram 2]FIG. 2 is a schematic top-down view of a monitored area in the exemplary environment of FIG. 1, the monitored area being located proximate a receiving / shipping portal and monitored by a video sensor of the order confirmation system of FIG. 1. [Diagram 3] FIG. 3 is a schematic diagram of an order confirmation system according to one embodiment of the present disclosure. [Figure 4] FIG. 4 is a block diagram illustrating the software architecture of the order confirmation system according to one embodiment of the present disclosure. [Diagram 5] FIG. 5 is a diagram of an example camera configuration for training data generation and subsequent detection of doors, including people and door states, according to one embodiment of the disclosure. [Figure 6] FIG. 6 illustrates detection of receiving / shipping portals having various degrees of closure / opening corresponding to closed, intermediate, and fully open states according to one embodiment of the disclosure. [Figure 7] FIG. 7 illustrates an example threshold for the height of a bounding box surrounding a receiving / shipping portal that can be used to determine whether the receiving / shipping portal is open or closed, according to one embodiment of the present disclosure. [Figure 8] FIG. 8 is a diagram illustrating an example pair of consecutive video frames from a portion of video footage, according to one embodiment of the present disclosure. [Figure 9] FIG. 9 illustrates an example Yolo v5 architecture that can be used to implement a palette detector, according to one embodiment of the present disclosure. [Figure 10] FIG. 10 illustrates a flow chart for keypoint detection according to one embodiment of the present disclosure. [Figure 11(a)] FIG. 11(a) is a virtual representation of a physical grid pattern marked on a ground area, according to one embodiment of the present disclosure. [Figure 11(b)] FIG. 11(b) is a virtual representation of the points from FIG. 11(a) projected by a camera according to one embodiment of the present disclosure. [Figure 12(a)]FIG. 12(a) illustrates an exemplary cube with opposing corner points T' and B', according to one embodiment of the present disclosure. [Figure 12(b)] FIG. 12(b) is a virtual representation of projections Tp and B corresponding to corner points T' and B' taken from the view of FIG. 12(a) according to one embodiment of the present disclosure. [Figure 13] FIG. 13 is a virtual representation of an object as seen by a camera placed above the object, according to one embodiment of the present disclosure. [Figure 14] FIG. 14 is a cross-sectional view of a pallet. [Figure 15] FIG. 15 is a cross-sectional view of a pallet with two boxes stacked on top of each other, the bottom box being longer than the top box.

[0013] In the accompanying drawings, underlined numbers are employed to represent the item in which the underlined number is located or to which it is adjacent. Non-underlined numbers refer to the item identified by the line linking the non-underlined number to the item. When a number is not underlined and is accompanied by an associated arrow, the non-underlined number is used to identify the general item toward which the arrow is pointing. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] The following detailed description illustrates embodiments of the disclosure and how they may be practiced. Although best modes of carrying out the disclosure are disclosed, those skilled in the art will recognize that there are other possible embodiments for carrying out or practicing the disclosure.

[0015] 1 and 2, the order confirmation system 110 comprises a number of video sensors 102 adapted to capture video footage of a monitored area located in the vicinity of the receiving / shipping portal 101. In one embodiment, the video sensors 102 may be attached to a frame 101a of the receiving / shipping portal 101. In another embodiment, the video sensors 102 may be positioned proximate to the receiving / shipping portal 101 such that the field of view (FOV) of the video sensors 102 includes at least the monitored area having either or both of the center and sides of the receiving / shipping portal 101, i.e., the area adjacent to the receiving / shipping portal.

[0016] The monitored area is thus located within an order receiving area or an order dispatching area. The monitored area is formed from the collective field of view (FOV) of the video sensors 102 and is bounded by the receiving / dispatch portal 101 and a region of interest (ROI) 204 (also known as a pallet analysis zone). The monitored area also comprises, in an embodiment of the present disclosure, a first buffer zone 202 and a second buffer zone 203 that are used together as a hysteresis decision function to eliminate uncertainty in determining whether a mobile entity is located outside or inside a corresponding order receiving area or order dispatching area.

[0017] In the illustrated example, the exterior monitoring zone 201 located to the left of the receiving / shipping portal 101 is considered to be outside the order receiving or order shipping area. Similarly, the area to the right of the receiving / shipping portal 101 is considered to be within the order receiving or order shipping area. Thus, in this example, the presence of an item approaching the order receiving area may be detected in the exterior monitoring zone 201. Furthermore, when an item is detected within the first buffer zone 202 and immediately thereafter, i.e., consecutively detected within the second buffer zone 203, the item is considered to have passed through the receiving / shipping portal 101 and entered the order receiving or order shipping area. Similarly, when an item is detected within the second buffer zone 203 and immediately thereafter, i.e., consecutively detected within the first buffer zone 202, the item is considered to have left the order receiving or order shipping area.

[0018] The peripheral portion of the order receiving or order dispatching area located near the receiving / dispatch portal 101 is represented by the internal remaining zone 205. The internal remaining zone 205 is not within the field of view of the video sensor 102 of the order confirmation system 110. Similarly, using the naming protocol of the present example, the internal remaining zone 205 is within the order receiving or order dispatching area. Thus, the internal remaining zone 205 is an unmonitored area within the order receiving or order dispatching area. Thus, the order confirmation system 110 of the present disclosure monitors the movement of entities (people, pallets) within the external monitoring zone 201, the first buffer zone 202 and the second buffer zone 203, and the region of interest (ROI) / pallet analysis zone 204, and thus, these aforementioned zones 201, 202, 203, and 204 together can be considered as a monitoring area for the sake of brevity in this disclosure.

[0019] 2, in a top-down view of the monitored area, the region of interest (ROI) / pallet analysis zone 204 and the first and second buffer zones 202 and 203 are defined as a set of rectangles. Those skilled in the art will understand that from a perspective view of the camera and corresponding to the setup presented in the diagram of FIG. 1, each of the region of interest (ROI) / pallet analysis zone 204 and the first and second buffer zones 202 and 203 are formed as trapezoids. Returning to the plan view of the monitored area shown in FIG. 2, the vertical dimension of the region of interest (ROI) / pallet analysis zone 204 and the first and second buffer zones 202 and 203 is equal to the vertical dimension of the receiving / shipping portal 101. Similarly, the first buffer zone 202 and the second buffer zone 203 have a horizontal dimension equal to the horizontal dimension of the receiving / shipping portal 101. In one example, the horizontal dimension of the region of interest (ROI) / pallet analysis zone 204 is configured to be three times the horizontal dimension of a standard goods pallet. However, those skilled in the art will recognize that the above relationships between the horizontal dimensions of the region of interest (ROI) / pallet analysis and the horizontal dimensions of a standard merchandise pallet are exemplary in nature and are provided for illustrative purposes only. In particular, those skilled in the art will recognize that the order confirmation system of the present disclosure is not limited to the above dimensional relationships. Rather, the preferred embodiment is operable with any horizontal dimensions associated with the inlet / outlet channels of an order receiving / order fulfillment facility.

[0020] 1 and 2, in one example, an entity 103 (e.g., a pallet of goods, shown below using the same numeral "103") traverses the receiving / shipping portal 101 in a left-to-right direction (moving from the exterior monitoring zone 201 to the interior remaining zone 205). The entity is considered to have entered the first buffer zone 202 when two conditions are met: (a) the entity is moving through the first buffer zone 202; (b) A box substantially, eg, digitally, representing the outline of the entity detected by the video sensor 102 of the order confirmation system 110 intersects the second buffer zone 203 .

[0021] For simplicity, the box that substantially represents the outline of the detected entity is hereafter referred to as the bounding box around the entity.

[0022] Referring to FIG. 3, the architecture of the order confirmation system 110 of the present disclosure includes: (a) a receiving / shipping portal having a plurality of video sensors 102; (b) a processing unit 301; (c) a database 302.

[0023] If an order receiving / order fulfillment facility has multiple receiving / shipping portals 101, a separate instance of the order confirmation system 110 may be dedicated to each receiving / shipping portal 101. In such an embodiment, some components may be shared between the separate instances of the order confirmation system 110, including, among others, the video sensor 102, the software detector components associated with the processing unit 301, and the database 302 of the order confirmation system 110.

[0024] The processing unit 301 comprises one or more CPUs, main memory, and local storage. The processing unit 301 is configured to run algorithms to detect entities, such as objects or people, in the video footage captured by the video sensor 102. The processing unit 301 is also configured to run algorithms to perform event analysis on the captured video footage. These algorithms are executed by a set of software detector components, described later in this specification and forming part of the order confirmation system 110. The database 302 stores information necessary to realize the algorithm execution / detector functions, as described below. Specifically, in various embodiments, the information stored in the database 302 includes: A dataset of face images / logos required for face / brand detectors, A dataset of product images required for product re-identification (run through the Pallet-by-Pallet Classification module described below).

[0025] The order confirmation system 110 of the present disclosure facilitates automated monitoring of order receiving or shipping areas by covering various aspects such as: - Detecting arrival of goods from third-party suppliers from door open events; - Identifying third-party suppliers and conducting a check-in process for delivery personnel from the third-party suppliers; - Correlating data on products actually received with those that suppliers should deliver (e.g., advanced shipping notices); -When detecting the entry and exit of goods via the receiving / shipping portal, verify that the detected delivered products match the data that should be delivered by the third-party supplier; -Ensure the outcome of the order confirmation process (approval / rejection of received orders) and recording of the courier's check-out at the end of the delivery.

[0026] Referring to FIG. 4, the software architecture of the order confirmation system 110 comprises three main software modules: a delivery detection module 402 , a pallet monitor module 404 , and an event (or alert) management module 406 .

[0027] The delivery detection module 402 is responsible for verifying whether the receiving / delivery process is performed correctly. The delivery detection module 402 includes a door status detector 402a, a person detector 402b, a person tracker 402c, and a quick response (QR) detector 402d. Based on the analysis of the video footage captured by the video sensor 102 of the order confirmation system 110, the door status detector 402a determines whether the receiving / shipping portal 101 is in an open or closed state. The person detector 402b analyzes the video footage captured by the video sensor 102 of the order confirmation system 110 to detect whether a delivery person has arrived at the receiving / shipping portal 101. Using the same video footage, the person tracker 402c tracks the movement of the delivery person detected by the person detector 402b. The QR detector 402d detects the presence of a quick response (QR) code in the captured video footage and reads the QR code. The QR detector 402d compares the detected QR code with known pre-approved QR codes for the third-party supplier / delivery person and finds a match. If a match is found, the person presenting the QR code is classified (i.e., the person is deemed by the QR detector 402d) as an authorized entrant to the order fulfillment / order receipt facility. Thus, if the movement of the detected person is tracked by the person tracker 402c and the person is deemed by the QR detector 402d as an authorized entrant, the delivery detection module 402 grants the person access to the order fulfillment / order receipt facility to perform the activities pursuant to the execution of the associated delivery.

[0028] The pallet monitor module 404 is dedicated to verifying the contents of goods delivered from or received into a premises such as an order processing / receiving facility. For this purpose, the first step is to detect the pallet using the pallet detector module 404a, then track the detected pallet using the pallet tracker module 404b, and finally classify the goods on the detected pallet using the goods per pallet classification module 404c. The pallet monitor module 404 also comprises a pallet volume estimator 404d, with the purpose of estimating the amount of goods on a pallet. The last component of the pallet monitor module 404 is an in / out counter 404e, which is used to extract information regarding the total number of pallets passing through the receiving / shipping portal.

[0029] The event (or alert) management module 406 comprises an alert manager 406a and an event recorder 406b. The alert manager 406a is configured to issue an alert on the detection of an authorized entrant and information about the goods being supplied / delivered, such as goods class, volume of the pallet, amount of pallets received during the supply / delivery episode in question, etc. The event recorder 406b is configured to record the entire supply / delivery episode. The above software components are described in more detail below.

[0030] The person detector 402b comprises a model used to detect the presence of authorized entrants, including but not limited to delivery personnel from third party suppliers and / or employees of the order fulfillment facility / order receiving facility, in the video footage captured by the video sensor 102 of the order confirmation system 110. The output from the person detector 402b is also processed by a person re-identification model (not shown) to track persons whose presence is detected in the captured video footage. The door status detector 402a comprises a model used to detect the presence of the receiving / shipping portal 101 in the captured video footage and a door status algorithm to determine whether the receiving / shipping portal is open or closed.

[0031] In one embodiment, the person detector 402b and the door status detector 402a may be combined into a software component. In this embodiment, a neural network is used to detect people and doors (i.e., receiving / shipping portal 101) based on the YOLO v5 architecture. The architecture selected is version M, which adds feature pyramid level P6 to the neck components of the original version. CSPDarknet53 (as described in C.-Y. Wang, H.-YM Liao, Y.-H. Wu, P.-Y. Chen, J.-W. Hsieh and I.-H. Yeh, CSPNet: A New Backbone that can Enhance Learning Capability of CNN, 2020 IEEE / CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2020, pp. 1571-1580) is the backbone of YOLO v5 used as a feature extractor. The neck is represented by a PANet (as described in S. Liu, L. Qi, H. Qin, J. Shi and J Jia, Path Aggregation for Instance Segmentation, 2018 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018, 8759-8768) for generating feature pyramids to help the model generalize at different scales. The head is used for final detection by generating anchor boxes and corresponding output vectors.

[0032] However, those skilled in the art will recognize that the above neural networks and architectures are exemplary in nature and are provided for illustrative purposes only. In particular, those skilled in the art will recognize that the order confirmation system 110 of the present disclosure is not limited to the above neural networks and architectures. Rather, the present disclosure can be implemented with any neural network and architecture capable of detecting people and objects, such as doors, in captured video footage. For example, the person detector 402b and the door state detector 402a can include a YOLO v5 architecture having an S-architecture or an L-architecture. Similarly, the person detector 402b and the door state detector 402a can comprise any single-shot detector (SSD), such as RetinaNet, or alternatively embody other types of neural networks and architectures known to those skilled in the art.

[0033] Further, in this embodiment, the person detector 402b and the door status detector 402a (or a combination of person and door status detector) are trained on a data set whose labels are door, employee, and delivery person. An exemplary camera setup for the generation of training data and subsequent person and door status detection is shown in FIG. 5. Those skilled in the art will appreciate that the order confirmation system 110 disclosed herein is not limited to the camera positions shown in FIG. 5. In particular, the cameras 501, 502, and 503 may be moved 5-10 cm in any direction from the positions shown in FIG. 5. The cameras 504 and 505 must be positioned such that the bottom of the receiving / shipping portal 101 is completely contained within the field of view of the cameras, and the entire pallet can be seen regardless of whether the pallet passes the receiving / shipping portal 101 on its right, left, or center.

[0034] Video footage of people in the training data is labeled according to the person's clothing. The labels assume that each operator at an order fulfillment or order receiving facility has a standard uniform that all employees must wear, where the uniform is easily distinguishable from clothing worn by non-employees. The labeling protocol also addresses the degree of uniform variation, e.g., all employees may wear brown shirts but have pants of a non-standard dark color (e.g., brown, black). Thus, using this approach, everyone wearing an operator uniform would be labeled as an "employee" and everyone else would be labeled as a "delivery person."

[0035] Exemplary details of the datasets used for training, validation, and testing (after being split into training, validation, and testing sets) of the order confirmation system 110 are as follows: Image size: minimum 1920x1080 pixels Number of annotated images: 18311 Number of different cameras (at least two viewpoints as shown in Figure 5) Total number of bounding boxes surrounding objects in video frames in the dataset: 59276 The number of bounding boxes surrounding objects of a particular class in video frames of the dataset: · Receiving / Shipping Portal: 16610 Delivery Person: 19461 Employees: 23205 During inference, the people detector 402b and the door state detector 402a (or a combination of people and door state detectors) receive as input images including video frames from video sensors 102, e.g., video footage captured by cameras 501-505 of the order confirmation system 110. In response to the received images, the people detector 402b (or a combination of people and door state detectors) outputs a 3D tensor including: (a) the coordinates of the center of a bounding box that encompasses a person or door detected in a received image; (b) a width and height of a bounding box, the width and height being normalized by scaling, respectively, to the width and height of the received image; and (c) an objectness score indicating the neural network's confidence, measured between 0 and 1, that an object center exists at a given location in the received image; and (d) Two output class predictions: “Employee” and “Delivery Person”.

[0036] A non-maximal suppression algorithm is used to generate the prediction with the highest confidence score from several overlapping bounding box proposals for the same person. The term "prediction" refers to the classifications of "employee" and "deliveryman" and the corresponding locations in the received image of the people so classified.

[0037] In addition to detecting people, the delivery detection module 402 also detects the presence of a receiving / shipping portal 101 in a received video frame and determines whether the receiving / shipping portal 101 is in an open or closed state. To this end, the YOLO v5 network of door status detector 402a (or a combination of people and door status detectors) generates an output classification of "door" upon detecting the presence of a receiving / shipping portal 101 in a received video frame. For a received video frame in which a receiving / shipping portal 101 is detected, a further output from the YOLO v5 network is a set of coordinates from which the height of a bounding box surrounding the detected receiving / shipping portal 101 may be calculated. With reference to FIG. 6, the bounding box heights, e.g., H1, H2, and H3, are used to determine the state of the receiving / shipping portal 101. Specifically, the receiving / shipping portal 101 is determined to be either in an open or closed state.

[0038] When the receiving / shipping portal 101 is closed, the height of the bounding box surrounding it has a maximum value. In contrast, ideally, when the receiving / shipping portal 101 is open, the height of the bounding box surrounding it is evaluated at 0, since the receiving / shipping portal 101 is no longer visible in the received video frames.

[0039] However, there are cases where the receiving / shipping portal 101 is not fully open. In this case, the threshold variable may be preset to a threshold value by an operator to avoid classifying the receiving / shipping portal 101 as closed. The threshold value may be empirically determined according to the environment and deployment in which the order confirmation system 110 is used. With reference to FIG. 7, if the height of the bounding box surrounding the receiving / shipping portal 101 is equal to or less than the threshold value HT, the receiving / shipping portal 101 is considered open. Otherwise, if the height value exceeds the threshold value HT, the receiving / shipping portal 101 is considered closed. Using this approach, it is recognized that the height H3 of the bounding box in FIG. 6 is equal to or less than the threshold value HT.

[0040] Returning to FIG. 4, a person tracker 402c is used to track a track path T ID, and assigns a unique ID for every person detected within an item of video footage and keeps a record of the unique ID. In one embodiment, the person tracker 402c performs tracking using a detection algorithm based on the DeepSort algorithm (as described in Wojke N., A. Bewley A. and Paulus D., “Simple online and realtime tracking with a deep association metric” 2017 IEEE International Conference on Image Processing (ICIP), Beijing, 2017, pp. 3645-3649). Specifically, the person tracker 402c uses the person detector 402b to establish bounding boxes around every person detected within every image of the captured video footage. In association with these bounding boxes, a unique ID is assigned to each detected person. Track path T ID ={(x1,y1),(x2,y2),…} represents a vector of spatial coordinates of the centers of bounding boxes corresponding to person IDs, stored in the order in which the bounding boxes are established in consecutive video frames. The positions are expressed in pixels in the frame coordinate system, whose origin is located in the top-upper corner of the video frame, with the OX axis running horizontally from the left to the right of the origin and the OY axis running vertically from the top to the bottom of the video frame.

[0041] Referring to FIG. 8, five people are detected in a first video frame captured at time T0, and five bounding boxes are established around the detected people in the first video frame. The five bounding boxes are assigned unique IDs, namely 180, 129, 159, 165, and 137, respectively. Immediately after the first video frame, in a second video frame captured at time T1, the same five people are visible, with the person with ID 137 partially obscured. Five bounding boxes are established around the people in the second video frame. The bounding boxes are each assigned a unique ID that corresponds to the location of the bounding box surrounding the same person appearing in the first video frame, even if the location of the bounding box in the second video frame is different from the location of the bounding box in the first video frame.

[0042] Those skilled in the art will appreciate that the unique IDs shown in the video frames of Figure 8 are provided for illustrative purposes. In particular, the people tracker of the order confirmation system is in no way limited to the use of these particular unique IDs or their particular values ​​as shown in Figure 8. Rather, the people tracker of the order confirmation system is operable with any unique ID that allows for identifying an individual between successive video frames of captured video footage and distinguishing an individual appearing in a video frame from other individuals.

[0043] In one embodiment, the DeepSort algorithm is used to perform the tracking. Every new person that enters the observed scene is assigned a new ID. For a person detected in a previous video frame, the DeepSort algorithm uses a representation of the person sufficient to recognize the same person if that person leaves and then re-enters the observed scene. Upon detecting and recognizing the person, the DeepSort algorithm assigns the person the same ID that was assigned when the person was detected in the previous video frame.

[0044] The DeepSort algorithm comprises a sort tracker and a re-identification (ReID) module implemented in a View Knowledge Distillation (VKD) (Porrello A., Bergamini L. and Calderara S., “Robust Re-identification by Multiple View Knowledge Distillation, Computer Vision”, ECCV 2020, Springer International Publishing, European Conference on Computer Vision, Glasgow, August 2020) neural network.

[0045] The original sort tracker (as described by Bewley A, Ge Z., Ott L., Ramos F. and Upcroft B., “Simple Online and Realtime Tracking”, 2016 IEEE International Conference on Image Processing (ICIP), Phoenix, AZ, 2016, pp. 3464-3468) tracks people from one video frame to another using only position and motion cues. The original sort tracker relies on the previous trackT of a person detected in the current video frame. ID and a Kalman filter module that receives the bounding box of the person's corresponding previous track T IDBased on the estimated position, the sort tracker estimates the location in the current video frame of the previously detected person. The sort tracker then compares the estimated location with the details of the bounding box surrounding each of the detected people in the current video frame to find its closest match. The measurement vector of the Kalman filter is represented by the size and location of the center of the bounding box. In addition, the state vector of the Kalman filter includes motion information (i.e., the derivatives of the measurement vector components). While simple and computationally efficient, the original sort tracker suffers from frequent identity switching in crowded scenes. To overcome this limitation, the information output from the DeepSort algorithm is integrated with the appearance information of the sort tracker extracted by a deep neural network trained offline. The neural network enables ReID, i.e., re-identification of people that have been detected previously but not seen for a while. The order confirmation system of the present disclosure uses a Views Knowledge Distillation (VKD) neural network to generate a better representation of the person's appearance for the purpose of ReID.

[0046] The VKD neural network learns numeric appearance descriptors for people. The appearance descriptors are trained such that the cosine distance between appearance descriptors from different poses of the same person is small and the cosine distance between appearance descriptors for different people is large. The VKD architecture consists of a Resnet feature extractor, such as Resnet50 or Resnet101, and a classification head. The appearance descriptor represents the smoothed output of the Resnet after applying global average pooling. The VKD architecture is trained with a classification loss applied to the classification head and a triplet loss applied to the appearance descriptor. VKD achieves improved performance by learning a robust representation using a teacher network and extracting knowledge into a student network. In this embodiment, the representation takes the form of an embedding vector of length 2048. However, one skilled in the art will recognize that the order confirmation system of the present disclosure is not limited to embedding vectors of this length. Instead, the order verification system disclosed herein is operable with embedding vectors of any length that allow for recognition of a person within the settings and environmental conditions of a given order processing facility / order receiving facility.

[0047] The Deep Sort algorithm uses both motion and appearance information to assign a unique ID to a person detected in a received video frame. For simplicity, a person assigned a unique ID is hereafter referred to as an enrollee. To enable tracking of an enrollee in subsequent received video frames by matching the enrollee with a person detected in the subsequent video frames, the Deep Sort algorithm retains the enrollee's ID along with the enrollee's appearance descriptor and the enrollee's corresponding position and motion information included in the corresponding Kalman filter state for a predefined number of subsequent received video frames.

[0048] If the enrollee does not match the person detected in a predefined number of subsequent received video frames, the enrollee's unique ID and corresponding appearance and movement information are discarded. In this embodiment, the predefined number of subsequently received video frames is 1000. However, those skilled in the art will recognize that the order confirmation system of the present disclosure is not limited to this number of subsequently received video frames. Instead, the order confirmation system disclosed herein can operate with any number of subsequently received video frames that allow recognition of a person who may have left the field of view of the video sensor of the order confirmation system and later re-entered the field of view, thereby meeting the requirements of the order receipt / order delivery process and the underlying conditions of a given order processing facility / order receipt facility.

[0049] The Jonker-Volgenant algorithm estimates the track T ID The Jonker-Volgenant algorithm is an efficient variant of the Hungarian algorithm. In the first phase, the Jonker-Volgenant algorithm is used to match the previous track T using information about the person's appearance. ID The algorithm then matches the current detection with the previously unmatched track. If after the first phase there are still unmatched current detections, the algorithm runs again in a second phase using the above motion information. After the second phase, the previous unmatched tracks are kept in the database for use in the next received video frame, and the current unmatched detections are used to create new tracks corresponding to some newly created IDs after a certain predefined warm-up period. In this embodiment, the warm-up period is three video frames. However, one skilled in the art will recognize that the order confirmation system of the present disclosure is not limited thereto, but rather, the number of frames used for the warm-up period may be empirically determined according to the environmental conditions and the settings of the order processing facility / order receiving facility.

[0050] In another embodiment, the Jonker-Volgenant algorithm matches the person detected in the currently received video frame with the track of the previously detected enrollee based on a weighted combination of a motion cost metric and an appearance cost metric. The motion cost metric may be calculated, for example, as the squared Mahalanobis distance between the Kalman filter measurement vector associated with the person detected in the current video frame and the measurement vector predicted by the Kalman filter of each previously detected enrollee. The appearance cost metric may be calculated, for example, as the cosine distance between the appearance descriptor of the person detected in the current video frame and the enrollee's appearance descriptor at each instance where the enrollee is detected in the previously received video frame.

[0051] Those skilled in the art will recognize that the above formulations for the motion cost metric and the appearance cost metric are provided for illustrative purposes only. In particular, those skilled in the art will recognize that the order confirmation system of the present disclosure is not limited to these above-mentioned formulations for the motion cost metric and the appearance cost metric. Instead, the order confirmation system disclosed herein is operable with any formulation of the motion cost metric and the appearance cost metric that supports matching a person detected in a currently received video frame with a previously detected enrollee. For example, the motion cost metric and / or the appearance cost metric may instead use a formulation that includes maximum likelihood statistics.

[0052] We define a newly enrolled person as the person to whom an ID was last assigned, and if the newly enrolled person does not match a person detected in a predefined number of subsequent video frames, the detection leading to the newly enrolled person is considered a false positive, and therefore the newly enrolled person's unique ID and corresponding appearance and movement information are discarded.

[0053] Sorting supports short-term matching of detected persons, while ReID supports long-term matching. Sorting involves hyperparameters that need to be tuned on a validation dataset that contains a sequence of video frames extracted from video footage captured by a video sensor of an order confirmation system at a constant frame rate, e.g., 4-7 frames per second. In contrast, the VKD algorithm employs a neural network trained on the ReID dataset. The ReID dataset contains regions cropped from the received video frames, where a cropped region corresponds to the area occupied by a bounding box containing a person. The cropped regions in the ReID dataset are also grouped into tracklets, each of which represents a region extracted from a video frame belonging to the received video footage.

[0054] The neural network used in the VKD algorithm can be trained or pre-trained using an open source dataset such as the Motion Analysis and Re-Identification (MARS). The ReID dataset used in the preferred embodiment has the following characteristics: Image size: variable (image of palette cropped using bounding box predicted by palette detector) Number of individuals / people: 48 Minimum number of bounding boxes per person: 30 Maximum number of bounding boxes per person: 3295 However, those skilled in the art will appreciate that both the values ​​associated with the above training / pre-training datasets and the above training / pre-training datasets are provided for illustrative purposes only. In particular, those skilled in the art will appreciate that the order confirmation system disclosed herein is not limited to using these datasets to train / pre-train a neural network with the VKD. Rather, the order confirmation system disclosed herein is operable with any dataset suitable for training / pre-training a neural network used in a VKD algorithm, including privately collected datasets.

[0055] Returning to Figure 4, the QR detector 402d implements a quick response (QR) detection algorithm. The purpose of the QR detector 402d is to enable identification of employees, such as delivery personnel or third party suppliers / buyers, based on the presence of a QR code on a tag attached to the person's uniform. In this manner, entry to the order fulfillment / order receiving facility can be controlled such that only authorized entrants, i.e., those who present the order fulfillment / order receiving facility with a tag bearing a QR code matching a known approved QR code of the supplier / delivery personnel, etc., are allowed access to the order fulfillment / order receiving facility.

[0056] In one embodiment, the QR detector 402d is implemented using a neural network based on the Yolo_v5 architecture, more specifically, Yolo_v5s. Yolo_v5 includes three main parts, namely, the backbone, the neck, and the head. The backbone employs a CSP-Cross Stage Partial Network, which is used to extract features from the input image / video frame. The neck is used to generate the feature pyramid. The neck includes the PANet, which helps the Yolo_v5s model generalize at different scales. The head is used for the final detection stage, specifically, the head generates the anchor boxes and the output vectors of the Yolo_v5s model. Those skilled in the art will recognize that the above network architectures are provided for illustrative purposes only. In particular, those skilled in the art will recognize that the order confirmation system 110 of the present disclosure is not limited to the use of the above network architectures. On the contrary, the order confirmation system 110 disclosed herein is operable with any suitable network architecture that enables the detection and recognition of QR codes present in an image. For example, the order confirmation system 110 may be operable with any other single shot detector (SSD), such as RetinaNet, previously disclosed herein.

[0057] A detailed example of the dataset used to train the Yolo_v5 network is provided below. Number of images: 1681 Image size: 2560x1440 pixels Comments: 1681 During training, a reference frame is created, which is a video frame taken from the captured video footage of the monitored area without the presence of a QR code. In a next step, a short video is cut from the raw video footage of the training dataset. The short video includes a series of video frames in which a QR code is displayed on a video camera. To ensure diversity of the feature distribution, the video frames are extracted from the short video using an average hashing algorithm. In one embodiment, the average hashing algorithm was implemented using the open source Python library ImageHash. However, it should be noted that the above-mentioned software tool for the average hashing algorithm is provided for illustrative purposes only. In particular, one skilled in the art will understand that the order confirmation system 110 of the present disclosure is not limited to the use of the ImageHash software tool. Rather, the order confirmation system 110 disclosed herein is operable with any software implementation of the average hashing algorithm.

[0058] In the average hash algorithm, a hash is calculated for each video frame in the short video. For simplicity, a given one of the second or subsequent video frames in the short video is hereafter referred to as a candidate QR image, and the video frame preceding the candidate QR image in the short video is hereafter referred to as a preceding candidate QR image. In an iterative process starting with the second video frame in the short video and proceeding stepwise through each remaining video frame in the short video, the hash of the candidate QR image is compared to the hash of the preceding candidate QR image and the hash of the reference frame. If the hash of the candidate QR image differs from the hash of the preceding candidate QR image by more than 5 values, the candidate QR image is selected and the hash of the preceding candidate QR image is updated with the hash of the candidate QR image. Similarly, if the hash of the candidate QR image differs from the hash of the reference frame by more than 7 values, the candidate QR image is selected and the hash of the reference frame is updated with the hash of the candidate QR image.

[0059] Once trained, the Yolo_v5 network of the above embodiment receives as input video frames from video footage captured by the video sensor 102 of the order confirmation system 110. In response, the Yolo_v5 network outputs three vectors as follows: (a) the coordinates of the center of a bounding box encompassing the QR code detected in the received image, together with a width and height of the bounding box, the width and height being normalized by scaling, respectively, relative to the width and height of the received video frame; (b) an objectness score indicating the neural network's confidence (evaluated between 0 and 1) that an object center exists at a given location in the received video frame; and (c) Class probability of detected objects. Upon detecting a QR code in a received video frame, a corresponding region is cropped from the video frame. The cropped region corresponds to a combination of the area of ​​the video frame occupied by a bounding box surrounding the QR code plus an additional 20 pixels added to each side of the bounding box to ensure that the entire QR code is contained within the cropped region. The QR code that appears in the cropped region is then decoded using a barcode reading software component. In one embodiment, the QR code reading software component is the Python library Pyzbar, which is based on the Zbar open source software suite. Those skilled in the art will appreciate that the barcode reading software components described above are provided for illustrative purposes only. In particular, those skilled in the art will appreciate that the order confirmation system 110 of the present disclosure is not limited to the use of the barcode reading software components described above. Instead, the order confirmation system 110 disclosed herein is operable with any software component capable of reading QR codes, such as, but not limited to, PyQRCode, qrcode, qrtools, etc.

[0060] 4 in conjunction with FIG 1, the output from the barcode reading software component includes a string decoded from a QR code detected in a received video frame. The delivery detection module 402 associates the string with a detected person that is closest to the QR code in the received video frame. Thus, the ability of the person tracker 402b to re-identify a person from one frame to another (based on appearance and motion attributes) is enhanced by the combination with the identity assigned to the person based on the QR code that they present to the video sensor 102 of the order confirmation system 110.

[0061] The palette detector module 404a implements a model that can detect, determine, and identify the location of palettes. In one embodiment, the palette detector module 404a implements a neural network based on the Yolo_v5 architecture, more specifically, Yolo_v5s, as shown in FIG. 9. Furthermore, as shown in FIG. 9, Yolo_v5 includes three main parts, namely, backbone, neck, and head. The backbone employs a Cross-Stage Partial (CSP) network that is used to extract features from the input image. The neck of the Yolo_v5 neural network is used to generate a feature pyramid. The neck includes a PANet, which helps the Yolo_v5 neural network generalize at different scales. The head is used for the final detection stage, generating anchor boxes and output vectors from the Yolo_v5 neural network.

[0062] Those skilled in the art will recognize that the above network architectures are provided for illustrative purposes only. In particular, those skilled in the art will recognize that the order confirmation system of the present disclosure is not limited to use of the above network architectures. Instead, the order confirmation system disclosed herein can operate with any suitable network architecture that allows for the detection and recognition of QR codes present in an image. For example, the order confirmation system disclosed herein can operate with any other single-shot detector, such as RetinaNet.

[0063] Exemplary details of the dataset used to train the Yolo_v5 network are as follows: Image size: 1920x1080 pixels Number of images (including palettes or parts of palettes taken at different angles): 1591 Number of bounding boxes enclosing palettes or parts of them in video frames of the dataset: 27873 Number of bounding boxes per class (the dataset should be balanced, i.e. each class should have the same number of bounding boxes):

[0064]

number

[0065] 1, 4 and 9, once trained, the Yolo_v5 network of the pallet detector module 404a receives as input video frames from video footage captured by the video sensor 102 of the order confirmation system 110. In response, the Yolo_v5 network outputs three vectors as follows: (a) the coordinates of the center of a bounding box encompassing the detected palette in the received image, together with a width and height of the bounding box, the width and height being normalized by scaling, respectively, relative to the width and height of the received video frame; (b) An objectness score that indicates the confidence, evaluated between 0 and 1, of a neural network that a palette center exists at a given position within the received video frame, and (c) The class probability of the detected palette.

[0066] Let the time at which a first video frame of a given item in a video is captured by a video camera, for example, the video camera 502 shown in FIG. 5, be time τ. The time interval Δt between captures of consecutive video frames of the video will hereinafter be referred to as the sampling interval. Using this notation, the video can be

[0067]

Number

[0068] described as.

[0069]

Number

[0070] denotes an individual video frame of the video, which is captured at time τ + iΔt and will hereinafter be known as the sampling time of the video frame.

[0071] For clarity, in the following disclosure, the current sampling time t k is given by t k = τ + NΔt, where N < n. The previous sampling time t p is the sampling time before the current sampling time t k and is given by t p = τ + DΔt, where 0 < D < N. The current video frame Fr(t k ) is the video frame captured at the current sampling time t k . The previous video frame Fr(t p ) is the previous sampling time tp 1 and 4, the currently detected palette is k ) detected by the palette detector module 404a. The previously detected palette is the palette detected by the palette detector module 404a in the previous video frame Fr(t p ) in the previous video frame Fr(t p ) by the palette detector module 404a. The current detection of the palette is k ) by the palette detector module 404a. Furthermore, the most recent previous detection of a palette is the closest previous sampling time to the current sampling time, i.e., for a given current time t k The most recent previous detection of a palette is one of one or more previous detections of a given palette by the palette detector module 404a in the previous video frame. The most recent previous detection of a palette is the last detection of the palette in the previous video frame.

[0072] The pallet tracker module 404b is communicatively coupled to the pallet detector module 404a and receives therefrom a list of pallets detected in the current video frame. The pallet tracker module 404b uses the output of the pallet detector module 404a to track the movement of pallets after they are detected. To this end, the pallet tracker module 404b tracks the center of each bounding box output from the pallet detector module 404a. Specifically, the pallet tracker module 404b processes video footage from all video sensors 102 of the order confirmation system 110 to track only pallets that traverse the receiving / shipping portal 101.

[0073] 2 and 4, it is noted that the area in which pallets are tracked includes several zones: an external monitoring zone 201, a first buffer zone 202, a second buffer zone 203, and a region of interest (ROI) / pallet analysis zone 204. To describe the path taken by each pallet as it moves within the area, each pallet tracked by the pallet tracker module 404b is assigned to a "track". A track has six associated attributes: (1) a unique track identifier (Tr_ID); and (2) A variable used to count the truck's lifetime, i.e., the time since the truck was last assigned to a pallet detected by the pallet detection module; and (3) A status variable indicating the status of the truck, i.e., the status variable indicates whether the truck is assigned to the pallet detected by the pallet detector module 404a, and the status variable can have one of two possible values, i.e., "assigned" and "unassigned." The default value of the status variable is "unassigned." (4) coordinates of a center of a bounding box encompassing the detected palette in the received video frame, together with a width and height of the bounding box, the width and height being normalized by scaling, respectively, to the width and height of the received video frame; (5) The code of the zone shown in Figure 2 (hereinafter referred to as the zone code), namely: 201-External Surveillance Zone 202 - First Buffer Zone 203 - Second Buffer Zone 204-Region of Interest (ROI) / Palette Analysis Zone 205-Inner Residual Zone (6) K path point vectors corresponding to each of the most recent K previous observations of the same palette.

[0074]

number

[0075] A path vector containing

[0076]

number

[0077] Each such path point vector then contains four attributes derived from observations of the pallet. Specifically, the attributes of a given path point include: A unique track identifier (Tr_ID) for the track to which the path vector is associated; and - the time when the corresponding previous observation of the pallet was made, and The coordinates of the center of the bounding box containing the palette of the corresponding previous look.

[0078] Therefore, at time t p The path point vector corresponding to a previous observation of a given pallet in

[0079]

number

[0080] where P = 0, 1, . . . , k-1.

[0081] The input to the palette tracker module 404b is a list of currently detected palettes, with each element in the list having the following attributes: The coordinates of the center of the bounding box that contains the currently detected palette, The corner coordinates of the bounding box that contains the currently detected palette, a zone code representing the zone (201, 202, 203, or 204 in FIG. 2) in which the currently detected pallet (as described by the center of the bounding box encompassing the currently detected pallet) is determined to be located by the pallet detector module 404b; and A palette flag indicating whether the currently detected palette is assigned to a track. The default value of the palette flag is FALSE. However, the palette flag may be updated to TRUE by the palette tracker module 404b once a track is identified to which the currently detected palette position may belong.

[0082] Using the above nomenclature, the current video frame Fr(t k ), the palette tracker module 404b uses the following method to match the currently detected palette with the track maintained by the palette tracker.

[0083] The default status of each track from the multiple tracks maintained by the pallet tracker is set to "unassigned."

[0084] The Euclidean distance is calculated between the center of the bounding box encompassing the currently detected pallet and the center of the bounding box surrounding each of the recently detected pallets whose track status variable has a value of "unassigned". The most recent previous detection of a pallet is indicated by the last element of the path vector of the track corresponding to the previously detected pallet in question. The currently detected pallet is assigned to the track of the previously detected pallet for which there is the smallest Euclidean distance between the most recent previous detection of that pallet and the currently detected pallet.

[0085] The palette flag of the assigned currently detected palette is set to true, and the status variable of the track in the problem is set to "assigned". Similarly, the center of the bounding box that contains the assigned currently detected palette is added to the end of the track's path vector. Thus, the track's path vector is increased in size by one path point vector that contains the following attributes: The unique track identifier (Tr_ID) of the track, The current sampling time, The coordinates of the center of the bounding box of the currently detected assigned palette.

[0086] The above sequence of processing steps is repeated for each currently detected pallet until there are no remaining tracks with the state variable "UNASSIGNED" or no currently detected pallets with the Palette Flag set to "FALSE" (in other words, no potential matching pairs remain). At the end of the process, if there are any currently detected pallets remaining that are not assigned to a track (i.e., there are any currently detected pallets remaining with the Palette Flag set to "FALSE"), a new track is created for the currently detected pallet.

[0087] The per-pallet product classification module 404c is configured to analyze the contents of the pallets. The per-pallet product classification module 404c comprises two communicatively coupled modules: an instance segmentation module that performs instance segmentation, and an image search module that uses an image search algorithm to classify cropped bounding boxes of products detected by the instance segmentation module.

[0088] Since the appearance of products often changes with the season and the year, instead of retraining the model for each new appearance of the class, it is more scalable to have a generic model that can detect the presence of the product and a further model for recognizing the product using prior knowledge in the form of an easily updateable product database. For this purpose, a model for detecting the presence of the product and a model for recognizing the product in the classes "pack", "box", and "vegetable" are trained. The classes can be further extended to include "small pack", "medium pack", and "large pack". Using this formulation, the details of a dataset that can be exemplarily used to train the model of the product classification module 404c by pallet are as follows: Image size: 1920x1080 pixels Number of images: 27050 Number of masks (annotations for different classes): 1114548 Number of masks per class: Staff: 24825 Vendor: 10565 Pack: 18400 Box:15501 Vegetables: 101 Flowers: 45 Fruit: 22 Instance segmentation is used because products on a pallet may be stacked irregularly and pixel-level masks improve the accuracy of product detection, which is the benefit of multi-task training. Bounding box-based detection and mask-based detection work synergistically to reduce errors.

[0089] In a preferred embodiment, the instance segmentation module employs a transformer-based model inspired by Swin (described in Z Liu, Y, Lin, Y. Cao, H. Yu, Y. Wei, Z. Zhang, S. Lin, and B. Gao, “Swin Transformer: Hierarchical Vision Transformer using Shifted Windows”, Proceedings of the IEEE / CVF International Conference on Computer Vision (ICCV), 2021, pp. 10012-10022). However, those skilled in the art will recognize that the above Swin-based transformer model is provided for illustrative purposes only. In particular, those skilled in the art will recognize that the order confirmation system 110 of the preferred embodiment is not limited to the use of the Swin transformer model. On the contrary, the order confirmation system 110 disclosed herein is operable with any transformer-based or CNN-based backbone that can be used for instance segmentation.

[0090] The image retrieval module executes an algorithm for product re-identification based on a neural network that learns embeddings of each instance of a product contained in the product image database. More specifically, the image retrieval module compares the visual appearance of the pallet in the received video frames with visual appearance information of products expected to be received / shipped by the order receiving / fulfilling facility based on, for example, an advanced shipping notice.

[0091] For example, consider vendor X, which has products "a", "b" and "c". For each of these products, the product database described above includes images that represent the current appearance of these products. From these images, information about the appearance of the products under various different conditions (e.g., from different viewpoints and rotation angles) can be represented as embedding vectors, which can be formed using embedding models such as VKD or Siamese Nets. Those skilled in the art will appreciate that these embedding models are provided for illustrative purposes only. In particular, those skilled in the art will appreciate that the order confirmation system 110 disclosed herein is not limited to the embedding networks described above. On the contrary, the order confirmation system 110 disclosed herein is operable with any encoder model that can form an embedding vector representation of the appearance of the products, such as a classical CNN that has been trained as a classifier and then had its head removed. To train the embedding model, several images (up to 10 images) of each product are provided. Furthermore, the VKD can be trained on images of the entire palette, rather than images of each product. However, for this approach, a fairly large number of images, e.g., at least 30 images, need to be provided.

[0092] The embedding vector forms a representation of the product (cola, chocolate, beer) that is robust to changes in appearance and viewpoint. This representation is used to identify the product in various images at various scales and positions, including various rotation and tilt angles between the product and the video sensor 102. The image search module of the pallet monitor module 404 compares the products detected in the received video frames to the products expected to be received / shipped by searching for the images and / or embedding vectors of the relevant products from a product database. The embedding vectors are used for search and / or re-identification via a simple distance metric in the embedding space. In one embodiment, the distance metric is a cosine metric. However, those skilled in the art will recognize that the above distance metric is provided for illustrative purposes only. In particular, those skilled in the art will recognize that the order confirmation system 110 of the present disclosure is not limited to the use of a cosine distance metric in the image search module. Rather, the order confirmation system 110 disclosed herein is operable with any suitable distance metric, such as a Euclidean distance metric.

[0093] If there are several products on a pallet, a representation of the palette can be constructed by combining the embeddings of all products visible on the pallet. Using this technique, it is not just possible to determine what products appear on the pallet. Instead, it is also possible to extract information that is useful for estimating the number of products in a pack and the number of packs on a pallet.

[0094] "Key points" are defined as points of interest on a pallet. Each key point corresponds to a corner of the pallet. Furthermore, the term "pallet" refers to the entire configuration of the wooden body and the products placed on the wooden body. In the preferred embodiment, 16 key points are defined, 8 for the wooden body of the pallet and 8 for the products stacked on the pallet. For the sake of brevity, the wooden body of the pallet and the products stacked on the pallet are hereafter collectively referred to as pallet subcomponents. Each of the above key points has a different class. The names of the classes consist of the pallet subcomponent name and a name that consists of a reference to each of the three axes, namely near, far, left, right, and up and down (e.g. product_far_left_top, product_far_left_bottom).

[0095] Pallets can be one of two types: Regular Pallet: A pallet having a rectangular parallelepiped shape, with the products stacked on it, is considered to be a single object and keypoints are annotated accordingly. Non-regular pallet: a pallet that does not have the shape of a rectangular parallelepiped (for example, when the shape of the product stack is not rectangular). In this case, the shape of the product stack is divided into multiple rectangular parallelepipeds.

[0096] 4, for non-regular palettes, the palette keypoint detector (not shown) of the palette monitor module 404 is configured to detect multiple keypoints of the same class. In contrast, for regular shaped palettes, the palette keypoint detector (not shown) detects unique keypoints (i.e., keypoints of different classes).

[0097] To this end, a palette keypoint detector (not shown) includes a convolutional neural network configured to receive cropped regions of a received video frame, the cropped regions including a palette. The palette keypoint detector (not shown) is configured to process the received cropped regions to generate 16 heatmaps, each heatmap including, for example, 128×128 pixels. Each heatmap determines the location of a corresponding keypoint within the cropped region.

[0098] Exemplary details of the dataset used to train the convolutional neural network are as follows: Image size: Variable, as the image for the palette is cropped using the bounding box established by the palette detector. Number of images: 2918 Number of comments: 30804 Number of annotations per class: Product_Far_Left_Top:2712 Product_Far_Right_Top:2777 Product_Far_Right_Bottom:791 Product_near_left_top: 2700 Product_near_left_bottom:2583 Product_near_right_top:2781 Product_near_right_bottom:2658 Palette_far_right_top: 752 Palette_far_right_bottom: 629 Palette_near_left_top:2543 Palette_near_left_bottom:2467 Palette_near_right_top:2640 Palette_near_right_bottom:2591 Palette_far_left_top:800 Palette_far_left_bottom: 596 Product_Far_Left_Bottom:784.

[0099] Thus, referring to FIG. 10, a palette keypoint detector (not shown) performs the following steps:

[0100] Detect palettes in a received video frame 1000. Palette detection is performed by a palette detector module as disclosed earlier herein.

[0101] From the received video frame, a region where the presence of a palette is detected by the palette detector module is cropped 1002. The cropped region corresponds to a combination of the area of ​​the video frame occupied by a bounding box surrounding the palette, plus an additional 20 pixels added to each side of the bounding box to ensure that the entire palette is included in the cropped region. For simplicity, this cropped region is hereafter referred to as the "cropped palette region." In practice, the cropped palette region may, for example, comprise 128x128 pixels, with the top left corner of the cropped palette region located at coordinates (x1, y1) of the received video frame.

[0102] The convolutional neural network sequentially processes 1004 each of the one or more cropped palette regions from the received video frame to generate one or more heatmaps. In one embodiment, the convolutional neural network is configured to generate 16 heatmaps from the cropped palette regions. However, those skilled in the art will recognize that the above number of heatmaps is provided for illustrative purposes only. In particular, those skilled in the art will recognize that the palette keypoint detector of the preferred embodiment is not limited to generating this number of heatmaps. Rather, those skilled in the art will recognize that the palette keypoint detector is operable to generate any number of heatmaps as needed to enable accurate detection of palette keypoints visible in the cropped palette regions from the cropped palette regions.

[0103] The multiple heatmaps are post-processed 1006 by a function to generate a corresponding number of lists of points, each of which corresponds to a palette keypoint.

[0104] The point is scaled 1008 to the dimensions of the crop palette region. So, for example, if the crop palette region has dimensions 128×128 and the point is defined by (x,y)=(0.46,0.76), then the point is located at approximately (x',y')=(59,97) in the crop palette region's coordinate system, whose upper left corner is denoted by (0,0).

[0105] The scaled points are transformed 1010 back into the coordinate system of the received video frame, the result of the transformation being the detected palette keypoints 1012.

[0106] Returning to FIG. 4, the palette volume estimation algorithm of the palette volume estimator 404d calculates the volume of objects by estimating their size from a 2D image. In general, it is not possible to recover a 3D position from a 2D projection, since an infinite number of points from a line in 3D space are projected to the same point on the 2D projection, i.e., on the camera plane. One possible solution to resolve the ambiguity exploited in stereo vision is to use a pair of views of the scene captured from different positions and recover the depth information using triangulation principles.

[0107] The preferred embodiment assumes a flat and horizontal floor and uses a homography to calculate the real-world coordinates of floor points from the camera coordinates. Figure 11(a) shows the points of a physical grid pattern marked on the ground, and Figure 11(b) shows the representation of these points on the camera projection. For simplicity, these grid pattern points are hereafter referred to as the reference grid. Using the correspondence of the corner points belonging to the same vertical line (e.g., the upper left corner T' of the front face of a cuboid corresponds to the lower left corner B' of the same face, as shown in Figure 12(a)) and the projection on the floor belonging to the basic geometry, the elevations of the parallelepiped corners from the ground are calculated. From these, the volume of the object is calculated.

[0108] The first step of the algorithm is to estimate the parameters of the homography transformation mapping point (x', y') from the floor to the camera pixel coordinates (x, y).

[0109]

number

[0110] Since not all parameters are independent, the homography matrix is ​​estimated up to scale. For this purpose, the matrix is ​​normalized. For example, in equation (1), h 22 can be set to a value of 1, so that the remaining eight parameters of the H matrix can be recovered from a set of four corresponding points with known locations, taken from a known reference grid. For better accuracy, more corresponding points with known locations are used.

[0111] Returning to FIG. 11, a reference grid is constructed with points marked on the floor, and homographies are estimated for several sets of four points. A least median squares robust estimation method is then used to find the solution parameters. In a possible embodiment, the spacing between the individual points of the grid (i.e., the grid size parameter d) is set to 50 cm.

[0112] In many cases, a pallet may be represented by a cuboid object. The volume of the cuboid object can be calculated using the size of its edges (in pixels). This is calculated from the key points representing the corners of the pallet. In particular, the width and length of the pallet are the size of the two adjacent edges resting on the floor. Thus, defining the bottom edge of the parallelepiped as its edge resting on the floor, the length of the bottom edge can be calculated from the corner of the pallet that corresponds to the parallelepiped. For this purpose, the location of the pallet corner is estimated by referring to the point from the ground pattern in Figure 11(a) that is observed to be closest to the corner. From this, the length of the bottom edge of the corresponding parallelepiped is calculated using equation (1) above.

[0113] The height of a rectangular object can be calculated as the distance between one corner resting on the floor and the opposite corner located directly above it. For example, referring to FIG. 13, T is located above B, so the projection T p B and C p The camera C observing the pallet corresponding to the parallelepiped is at a known height C above the floor. height It is placed at height C height is the length of Figure 13 |CC p |is.

[0114]

number

[0115] and

[0116]

number

[0117] also represents the height. Therefore,

[0118]

number

[0119] and

[0120]

number

[0121] is perpendicular to the floor and aligned parallel to it. Therefore, equation (2) defines the triangle T p C p C and triangle T p This can be established from the similarity between the BT triangle.

[0122]

number

[0123] Sometimes, a pallet may exhibit a shape different from a rectangular prism. Two common cases are when (a) the number of items on different pallet rows differs from each other, and (b) the pallet consists of a set of non-uniform packs of different items, each of which exhibits a rectangular prism shape.

[0124] For objects of different shapes (e.g., case (a) disclosed above), the line CC in FIG. p Since the above method can accurately determine only the points that correspond to the line perpendicular to the floor (representing the height), additional key points are needed for volume calculation. For example, in Fig. 14, point D, which is located below point A, is needed for the estimation of the position of point A. Otherwise, the area of ​​surface ABCE would be erroneously estimated when |EA| is the slant height of the object shown in Fig. 14. Specifically, point A is located at CC in Fig. 13. p and would be estimated to be higher than the floor because the pixel distance between points A and E is greater than the pixel distance between points A and D. In order to accurately estimate the area of ​​surface ABCE, the areas of ABCD and ADE need to be calculated separately.

[0125] Figure 15 shows a different pallet shape representative of case (b) disclosed above, where the pallet is composed of a non-uniform set of packs of different items. In this case, the pallet shape is formed of two rectangular objects stacked on top of each other. Point G can be calculated using the above method only if point G' below it is known (given as a key point). Similarly, point E can be estimated since EG' is perpendicular to the floor.

[0126] Thus, in each received video frame, keypoints representing the corners of the palette visible therein are detected by a palette keypoint detector (not shown), as well as other useful points (e.g., G' as shown in Figure 15). Returning to Figure 4, the palette volume estimation algorithm (not shown) of the palette volume estimator 404d is configured to use the above Apache to calculate the volume of the palette in each of the video frames if enough keypoints (palette corners) are visible and detected, regardless of the angle of the palette relative to the camera observing it.

[0127] 4 in conjunction with FIG. 1, the in-out counter 404e is configured to use a path determined by the pallet tracker module 404b and defined as a sequence of the form:

[0128]

number

[0129] , where k=0,1,...,K, and detects which paths intersect with the receiving / shipping portal 101, thereby determining which pallets are entering or leaving the order fulfillment facility / order receiving facility. Using this information, the order confirmation system 110 records incoming and outgoing pallets (identified by their IDs) along with their time of entry / exit. Additionally, a count may be kept of the number of inbound / outbound pallets to / from the order fulfillment facility / order receiving facility over a given period of time (as determined by the direction of pallet movement).

[0130] Additionally, a value of a pallet status variable may be recorded for the pallet. The pallet status variable characterizes the degree to which the pallet is loaded. For example, the value of the pallet status variable may be "full", "empty", "partially loaded", etc. The value of the pallet status variable may be determined using the height of the goods stacked on the pallet as determined by the pallet volume estimator 404d. Knowing the maximum allowable stack height of a pallet, the pallet may be classified as follows: - Fully loaded if the height of the stacked goods is close to the maximum permitted stacking height, or - Partial load, if the height of the stacked goods is approximately half the maximum permitted stacking height, or - A pallet is empty if it has no products on it.

[0131] An event (or alert) management module 406 triggers specific alerts specific to individual applications based on events of interest (e.g., exceeding a maximum door open time, invalid access to a receiving / shipping portal, etc.). Events are generated, for example, using outputs from door status detector 402a, people detector 402b, and QR detector 402d. Events of interest are also recorded as GIF files and saved to disk for later use.

[0132] The event (or alert) management module 406 executes logic for a particular alert, such as, but not limited to: The receiving / shipping portal 101 remains open for a period longer than a certain threshold; and A pallet remains in a particular area for a period longer than a certain threshold, - The contents of pallets from certain vendors do not match the advance shipping notices; Pallets that leave the receiving / shipping portal 101 without being registered with the order confirmation system 100; The height of the pallet exceeds a certain maximum allowable height, - No staff from the order fulfillment / receiving facility is present when the delivery person arrives; · A delivery person entering an order fulfillment / receiving facility without being signed in.

[0133] These and other types of alerts are based on the output from, for example, the door status detector 402a, the people detector 402b, and the QR detector 402d. For example, when the receiving / shipping portal 101 is opened, a timer is started and when the timer reaches a certain threshold, an alert is triggered and the event is logged.

[0134] When an alert is triggered, the event recorder 406b is configured to save a set of consecutive video frames on disk (consecutive video frames can also be assembled and packed as an animated GIF file), starting from a specific period before the alert and ending from a specific period after the alert (for example, the period can be extended from 10 seconds to 60 seconds based on the type of alert). The generated video frames / GIFs are stored and can be reviewed by staff at any time. The maximum number of stored video frames / GIFs and the period they are kept can be configured according to the operator's needs and are handled by the predefined logic of the event recorder 406b based on application-specific requirements.

[0135] Modifications to the embodiments of the present disclosure described above are possible without departing from the scope of the present disclosure, as defined by the appended claims. Words such as "including," "comprising," "incorporating," "consisting," "having," "is," and the like, used to describe and claim the present disclosure, are intended to be interpreted in a non-exclusive manner, i.e., to take into account the presence of items, components, or elements not expressly described. References to the singular should also be interpreted as relating to the plural.

Claims

1. a plurality of video sensors configured to capture video footage of a monitor area located within an order receiving area or an order shipping area of ​​the receiving / shipping portal; performing an event analysis on the captured video footage; Detecting an entity within the video footage captured by the video sensor; Detecting a shipment from a third-party supplier from a door opening event in the captured video footage; Identifying the third-party supplier and conducting a check-in process for a delivery person from the third-party supplier; a processing unit configured to detect the entry and exit of goods through the receiving / shipping portal and to verify that the detected delivered products match data regarding products to be delivered by the third-party supplier; a database communicatively coupled to the processing unit, the database comprising: At least one dataset of face images / logos for use in face / brand detection; A dataset of product images for use in product identification; and said database configured to store, upon completion of delivery, a result of the order confirmation process and a record of the delivery person's check-out for future retrieval upon request to said processing unit; An order confirmation system comprising:

2. The processing unit includes: a door status detector configured to analyze the video footage captured by the video sensor and determine whether the receiving / shipping portal is in an open or closed state; a people detector configured to analyze the video footage captured by the video sensor of the order confirmation system and detect whether a delivery person has arrived at the receiving / shipping portal; a person tracker configured to track a movement of the delivery person through the captured video footage upon detection of the delivery person by the person detector; 10. The order confirmation system of claim 1, further comprising a delivery detection module comprising: a quick response (QR) detector configured to detect the presence of a QR code in the captured video footage and read the QR code.

3. The QR detector compares the detected QR code with known pre-approved QR codes for third party suppliers / delivery personnel to find a match, and if a match is found, the QR detector: classifying the delivery person who presents the QR code as an authorized entrant to an order fulfillment / order receiving facility; The order confirmation system of claim 2 , wherein the delivery detection module is used to grant the delivery person access to the order processing / order receiving facility.

4. 3. The order confirmation system of claim 2, wherein the person detector and the door status detector are embodied as neural networks of a predetermined architecture configured for person and door detection.

5. The processing unit includes: and a pallet monitor module configured to verify one or more contents of goods delivered or received from the premises of the order fulfillment facility / order receiving facility, the pallet monitor module comprising: a pallet detector module configured to detect a pallet; a pallet tracker module configured to track the detected pallet; a per-pallet item sorting module configured to sort items on the detected pallets; a pallet volume estimator configured to estimate a quantity of items on the detected pallet; 3. The order confirmation system of claim 2, further comprising: an in-out counter configured to extract information regarding a total number of pallets passing through the receiving / shipping portal.

6. The processing unit further comprises an event management module in communication with the delivery detection module and the pallet monitor module, the event management module comprising: an alert manager configured to issue an alert regarding the detection of an authorized entrant and information regarding the goods being supplied / delivered during a supply / delivery episode; an event recorder configured to record said supply / delivery episodes; 6. The order confirmation system of claim 5, comprising:

7. The alert manager when the receiving / shipping portal remains open for a period of time that exceeds a first predefined threshold; when the pallet remains in a particular region for a period of time that exceeds a second predefined threshold; when the contents of the pallet from a particular third-party supplier do not match the corresponding details in the Advanced Shipping Notice; When the pallet leaves the receiving / shipping portal without being pre-registered, when the height of the pallet exceeds a predefined maximum allowable height; When the delivery person arrives and no employees of the order processing facility / order receiving facility are present, 7. The order confirmation system of claim 6, further configured to issue an alert when the delivery person enters the order processing / order receiving facility without first signing in.

8. 1. A method for performing video surveillance, the method comprising: capturing video footage of a monitor area located within an order receiving or order shipping area of ​​a receiving / shipping portal using a plurality of video sensors; performing an event analysis on the captured video footage using a processing unit; detecting, using the processing unit, an entity within the video footage captured by the video sensor; detecting, using the processing unit, an arrival of a package from a third party supplier from a door opening event in the captured video footage; using the processing unit to identify the third-party supplier and perform a check-in process for a delivery person from the third-party supplier; using said processing unit to detect the entry and exit of goods through said receiving / shipping portal and verify that the detected delivered products match data regarding products to be delivered by said third-party supplier; using a database to store at least a dataset of face images / logos for use in face / brand detection and a dataset of product images for use in product identification; using said database to record the result of the order confirmation process and the check-out of the delivery person upon completion of the delivery; retrieving said record by said processing unit from said database in response to a request to said processing unit; The method includes:

9. analyzing the video footage captured by the video sensor to determine whether the receiving / shipping portal is in an open or closed state; analyzing the video footage captured by the video sensor of the order confirmation system to detect whether a delivery person has arrived at the receiving / shipping portal; upon detecting a delivery person, tracking the movement of the delivery person through the captured video footage; detecting the presence of a Quick Response (QR) code within the captured video footage; reading the detected QR code as an output; The method of claim 8 , further comprising:

10. Further comprising the step of comparing the detected QR code with known pre-approved QR codes for third party suppliers / delivery personnel to find a match, and if a match is found in the event, classifying a delivery person presenting said QR code as an authorized entrant to the order fulfillment / order receiving facility; The method of claim 9 , further comprising granting the delivery person access to the order fulfillment / order receiving facility.

11. 10. The method of claim 9, further comprising the step of: executing a neural network of a predetermined architecture configured for person and door detection.

12. The method further includes the step of verifying one or more items of merchandise to be delivered or received from the premises of the order fulfillment facility / order receiving facility, the verifying step comprising: detecting a pallet using a pallet detector module; tracking the detected pallets using a pallet tracker module; classifying items on the detected pallets using a pallet-by-pallet item classification module; estimating a quantity of items on the detected pallet using a pallet volume estimator; and extracting information regarding a total number of pallets passing through the receiving / shipping portal using an in-out counter.

13. issuing an alert upon detection of an authorized entrant and information regarding the goods being supplied / distributed during a supply / delivery episode using an alert manager; The method of claim 12 further comprising the step of: recording said supply / delivery episode using an event recorder.

14. the receiving / shipping portal remains open for a period of time that exceeds a first predefined threshold; the pallet remains in a particular region for a period of time that exceeds a second predefined threshold; the contents of the pallet from a particular third-party supplier do not match the corresponding details in an advanced shipping notice; the pallet leaves the receiving / shipping portal without being pre-registered; the height of the pallet exceeds a predefined maximum allowable height; When the delivery person arrives, no employees of the order fulfillment facility / order receiving facility are present; 14. The method of claim 13, further comprising the step of issuing an alert by said alert manager in the event said delivery person enters said order fulfillment / order receiving facility without previously signing in.

15. When executed by a processing unit, the processing unit: capturing video footage of a monitor area located within an order receiving or order shipping area of ​​a receiving / shipping portal using a plurality of video sensors; performing an event analysis on the captured video footage; and detecting an entity within the video footage captured by the video sensor; detecting an arrival of a package from a third-party supplier using the processing unit from a door opening event in the captured video footage; identifying said third-party supplier and performing a check-in process for a delivery person from said third-party supplier; detecting the entry and exit of goods via said receiving / shipping portal and verifying that the detected delivered products match data regarding products to be delivered by said third-party supplier; using a database to store at least a dataset of face images / logos for use in face / brand detection and a dataset of product images for use in product identification; Using said database, upon completion of delivery, record the result of the order confirmation process and the check-out of the delivery person; retrieving said records from said database in response to a request by said processing unit; A non-transitory computer-readable medium having stored thereon computer-executable instructions for causing a computer to execute a program.

16. Upon execution of the executable instructions, the processing unit: analyzing the video footage captured by the video sensor of the order confirmation system to determine whether the receiving / shipping portal is in an open or closed state; analyzing the video footage captured by the video sensor to detect whether a courier has arrived at the receiving / shipping portal; Upon detecting a delivery person, tracking the movement of the delivery person from the captured video footage; 16. The non-transitory computer-readable medium of claim 15, configured to detect the presence of a Quick Response (QR) code in the captured video footage and read the detected QR code as an output.

17. Upon execution of the executable instructions, the processing unit: configured to compare the detected QR code with known pre-approved QR codes for third-party suppliers / delivery personnel to find a match, and if a match is found in the event, classifying a delivery person presenting said QR code as an authorized entrant to the order fulfillment / order receiving facility; 20. The non-transitory computer readable medium of claim 15, further comprising: granting the delivery person access to the order fulfillment / order receiving facility.

18. Upon execution of the executable instructions, the processing unit: configured to verify one or more items of merchandise to be delivered or received from the premises of said order fulfillment facility / order receiving facility, said verifying comprising: detecting a pallet using a pallet detector module; tracking the detected pallets using a pallet tracker module; classifying items on the detected pallets using a pallet-by-pallet item sorting module; estimating a quantity of items on the detected pallet using a pallet volume estimator; and extracting information regarding a total number of pallets passing through the receiving / shipping portal using an in / out counter.

19. Upon execution of the executable instructions, the processing unit: issuing alerts regarding detection of authorized entrants and information regarding said goods being supplied / delivered during a supply / delivery episode using an alert manager; 20. The non-transitory computer-readable medium of claim 18 configured to record the supply / delivery episodes using an event recorder.

20. Upon execution of the executable instructions, the processing unit: the receiving / shipping portal remains open for a period of time that exceeds a first predefined threshold; the pallet remains in a particular region for a period of time that exceeds a second predefined threshold; the said contents of a pallet from a particular third-party supplier does not match the corresponding details in the advance shipping notice; the pallet leaves the receiving / shipping portal without being pre-registered; the height of the pallet exceeds a predefined maximum allowable height; When the delivery person arrives, no employees of the order fulfillment facility / order receiving facility are present; 20. The non-transitory computer-readable medium of claim 18, configured to issue an alert using the alert manager in the event the delivery person enters the order fulfillment facility / order receiving facility without first signing in.

Images