Movable shelf system based on body recognition

Abstract

To provide a movable shelf system for a retail or grocery store.SOLUTION: The system includes: a movable shelf in a store location; an actuator for moving the movable shelf; a plurality of cameras in the store location, each of the cameras having a field of view; and a vision mesh network having a plurality of nodes in communication with each other, at least one of the cameras being a node of the plurality of nodes on the vision mesh network. The vision mesh network receives and processes information about the movable shelf and sends an instruction to the movable shelf based on the information.SELECTED DRAWING: Figure 4A

Description

[Background technology]

[0001]

[0001] In retail locations, such as grocery stores, customers may be unable to access products on shelves because the shelves are too high or the products are out of reach. Customers, especially those with disabilities, may face difficulty and inconvenience when they cannot reach products on shelves without the assistance of others.

[0002]

[0002] The present disclosure will be described with reference to the following figures: [Brief explanation of the drawings]

[0003] [Figure 1]

[0003] FIG. 1 is a plan view of a store location having a movable shelving system according to some embodiments of the present disclosure. [Figure 2]

[0004] 1 illustrates an aisle having an adjustable shelving system according to some embodiments of the present disclosure. [Figure 3] FIG. 1 is a floor plan view of a store location having a movable shelving system according to some embodiments of the present disclosure. [Figure 4A]

[0005] 1A-1C illustrate a customer interacting with an adjustable shelving system according to some embodiments of the present disclosure. [Figure 4B] 1A-1C illustrate a customer interacting with an adjustable shelving system according to some embodiments of the present disclosure. [Figure 4C] 1A-1C illustrate a customer interacting with an adjustable shelving system according to some embodiments of the present disclosure. [Figure 5A]

[0006] 1A-1C illustrate a customer in an electric shopping cart interacting with an adjustable shelving system according to some embodiments of the present disclosure. [Figure 5B] 1A-1C illustrate a customer in an electric shopping cart interacting with an adjustable shelving system according to some embodiments of the present disclosure. [Figure 5C]1A-1C illustrate a customer in an electric shopping cart interacting with an adjustable shelving system according to some embodiments of the present disclosure. [Figure 6A]

[0007] 1 illustrates a number of customers interacting with an adjustable shelving system according to some embodiments of the present disclosure. [Figure 6B] 1 illustrates a number of customers interacting with an adjustable shelving system according to some embodiments of the present disclosure. [Figure 7]

[0008] 1 illustrates an adjustable shelf system according to some embodiments of the present disclosure. [Figure 8]

[0009] FIG. 1 is a flow diagram illustrating an adjustable shelf system according to some embodiments of the present disclosure. [Figure 9]

[0010] FIG. 1 is a flow diagram illustrating an adjustable shelf system according to some embodiments of the present disclosure. [Figure 10]

[0011] FIG. 1 is a basic block diagram of a data processor that can be used to process data provided over a vision mesh network according to some embodiments of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0004]

[0012] Exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. However, the present disclosure may be embodied in many different forms and should not be construed as limited to the specific exemplary embodiments set forth herein. Rather, these exemplary embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0005]

[0013] When a customer is shopping at a retail or grocery store and cannot reach or otherwise access an item on a shelf or display, the customer must ask for assistance. Often, this assistance is provided by another customer or a store employee. The customer who asks for help is forced to wait before assistance is provided. The wait time or inconvenience of requesting help multiple times or for multiple items, for example, can negatively impact the customer's shopping experience. Customers may also feel hesitant or uncomfortable requesting help.

[0006]

[0014] Retail and grocery stores typically include multiple cameras installed throughout the store to monitor employees, customers, loss prevention, inventory, security, etc. The cameras may be smart cameras and / or edge cameras and may be positioned to provide a viewing angle of a desired target area of ​​the store. Target areas may include, for example, point-of-sale areas such as checkout and self-checkout stations, entrances, exits, aisles, displays, pharmacy areas, etc. Target areas may encompass the entire store and may include overlapping areas and fields of view. The cameras may be connected to each other to form a vision mesh network.

[0007]

[0015] As used herein, "edge camera" generally refers to a camera that features on-board storage. Edge cameras can be used as security cameras that store recorded video footage data on their internal storage hardware. Thus, recording at the edge refers to recording video and storing it either on the camera or on a directly attached storage device, rather than transporting it over a network to a centralized recording mechanism, such as a digital video recorder.

[0008]

[0016] According to some embodiments of the present disclosure, a "vision mesh network" directly connects cameras throughout a store. The cameras recognize customer behavior, including movements and actions, and can provide convenience to customers. In one example, an edge camera can recognize when a customer is pointing at an item on a shelf that is out of the customer's reach and command the shelving unit to lower the shelf, and recognize when a customer selects the item and command the shelf to return to its original position.

[0009]

[0017] As used herein, a "vision mesh network" refers to a network that acts as a single network of cameras, e.g., a group of cameras or multiple cameras connected via a network including a Wi-Fi router. Thus, instead of just one camera, there are multiple data sources. By providing access to each edge camera throughout the store, images or data from each edge camera can be used as data input for the adjustable shelving system. This vision mesh network can be accessed ad hoc to determine whether there are useful, additional, or alternative views of a target area that can be used as additional data input. Information is shared within the vision mesh network, allowing a single camera to make decisions about customers or activities, such as whether a customer is interacting with the adjustable shelving system. Input from one, some, or all of the cameras in the vision mesh network can be utilized. Increasing the quantity and quality of data inputs entering the computer vision module to determine the correct self-checkout station operation or checkout procedure will improve the accuracy and reliability of the system. While it is possible to centralize all camera data streams through a client-server architecture, centrally process all inputs, and distribute the results accordingly, issues such as data security, increased network resources, and increased latency can significantly impede that type of solution.

[0010]

[0018] As used herein, a computer vision module includes methods for acquiring, processing, analyzing, and understanding digital images and extracting high-dimensional data from the real world to form decisions. Various embodiments of a computer vision module described herein acquire, process, analyze, and understand images and data from a vision mesh network to provide feedback and operational decisions regarding the customer, the shopping experience, and / or the adjustable shelving system. In some embodiments, the computer vision module includes a body recognition module and a face recognition module that can recognize a customer's body movements and facial features, respectively, to identify and authenticate the customer, determine if the customer needs assistance, and assist the customer when an item is out of reach. In further embodiments, these modules can see how a customer behaves while shopping and when assistance is needed.

[0011]

[0019] 1 and 2 , a mobile shelving system 10 in a store location 18, including, for example, a mobile shelf 32d, according to some embodiments of the present disclosure, will be described. As illustrated, the store location 18 includes multiple aisles 12, 14, 16 defined by shelf sets 20, 22, 24, 26, 28 adjacent to at least one side of the aisles 12, 14, 16. The shelf sets 20, 22, 24, 26, 28 include multiple individual shelving units 30, 32, 34, 36 connected together, positioned together, and / or installed in groups or clusters near or next to each other to form the shelf sets 20, 22, 24, 26, 28. The shelving units 30, 32, 34, 36 may be freestanding. The shelving units also include end cap shelving units 31 located at the ends of the aisles or shelf sets.

[0012]

[0020] As illustrated in FIG. 2 , the aisle 12 is lined with shelf sets 20 and 22 on the left and right sides, respectively. The shelf sets 20 and 22 include multiple separate shelving units 30, 32, 34, and 36 arranged together to form the shelf sets 20 and 22, defining the desired length of the aisle 12. Each shelving unit 30, 32, 34, and 36 includes at least one shelf, and in one example, multiple shelves 30a, 30b, 30c, 30d, 32a, 32b, 32c, and 32d. At least one shelf is an adjustable shelf 32d according to one embodiment of the present disclosure. The adjustable shelf 32d is a “smart shelf” that is programmably controlled and remotely operated. As illustrated in FIG. 2 , the adjustable shelf 32d is the top shelf of the shelving unit 32.

[0013]

[0021] Any number of adjustable shelves 32d may be provided. The number and arrangement of aisles 12, 14, 16, shelf sets 20, 22, 24, 26, 28, shelving units 30, 32, 34, 36, end caps 31, shelves 30a, 30b, 30c, 30d, 32a, 32b, 32c, and adjustable shelves 32d can be adjusted and customized as desired. In some preferred embodiments of the present disclosure, each individual shelving unit 30, 32, 34, 36 and end cap includes adjustable shelves. In one example, most or all of the shelves are adjustable. When a customer selects items from shelves 30a, 30b, 30c, 30d, 32a, 32b, 32c, and adjustable shelves 32d, the customer passes through aisles 12, 14, 16 defined by shelf sets 20, 22, 24, 26, 28.

[0014]

[0022] As illustrated in FIGS. 1-3 , the store location 18 includes at least one edge camera 40. Each edge camera 40 has an associated corresponding field of view (FOV) 42. The edge cameras 40 are mounted such that the FOV 42 includes a desired target area. The edge cameras 40 may be positioned such that the FOVs 42 overlap one another and / or form a complete view of the store location 18. The edge cameras may also be positioned such that the FOV 42 is focused on a selected target area, such as adjustable shelving 32d. In some embodiments of the present disclosure, the edge cameras 40 are mounted throughout the store location 18, allowing the edge cameras 40 to monitor customers as they shop and, through computer vision, including facial and body recognition, to recognize when customers need assistance while shopping.

[0015]

[0023] Each edge camera 40 is a node on the vision mesh network 50. Each edge camera 40 is connected to at least one other edge camera 40 or to nodes that form the vision mesh network 50. In one example, each edge camera is directly connected to all other cameras 40. Each edge camera 40 may be connected to any number of other edge cameras 40 as desired. The edge cameras 40 are connected to each other wired or wirelessly, or any combination thereof. Using a wired setup may reduce latency and interference. The edge cameras 40 may be movable or fixed, or any combination thereof. It will be understood that some or all edge cameras 40 may be directly connected to each other on the vision mesh network without departing from the scope of this disclosure. Additional devices may serve as nodes on the mesh network. In one example, further described below with reference to FIGS. 4A-4C , the controller 68 for the adjustable shelf 32d is a node on the vision mesh network 50.

[0016]

[0024] Each edge camera 40 processes information close to the source of the data, i.e., at the edge camera 40, and calculations, including analysis of images and other data, are performed on the edge camera 40. Edge computing reduces the likelihood of having to send images and other data to a central server or cloud service for processing, which can increase processing speeds and reduce stress on the overall network.

[0017]

[0025] Each edge camera 40 has a designated area of ​​responsibility. The edge cameras 40 provide data input, such as images or video capture, to the designated edge camera, which processes the data input through a computer vision module, which may include a facial recognition module and a body recognition module, and outputs an operational decision based on the input data. The accuracy of the module is directly related to the quality and quantity of the input data. As described above, the computer vision module includes methods for acquiring, processing, analyzing, and understanding digital images and extracting high-dimensional data from the real world to form decisions. Various embodiments of the computer vision module described herein acquire, process, analyze, and understand images and data from the vision mesh network to provide feedback and operational decisions regarding the customer, the shopping experience, and / or the adjustable shelving system. In some embodiments, the computer vision module includes a body recognition module and a face recognition module that can recognize a customer's body movements and facial features, respectively, to identify and authenticate the customer, determine if the customer needs assistance, and assist the customer when an item is out of reach. For example, computer vision modules can watch for gestures indicating a person is trying to reach a shelf that is inaccessible because the person is short, a child, and / or a person with a disability. The computer vision modules can look for behaviors indicative of a request for assistance, including pointing, gesturing, or reaching toward an item. In further embodiments, these modules can see how customers behave while shopping and when they need assistance. For this reason, multiple cameras 40 are installed throughout the store location 18. Target areas can include, for example, adjustable shelving 32d and customers who have registered to receive assistance, for example, through a loyalty program or other opt-in program.

[0018]

[0026] The vision mesh network 50 of Figure 3 connects edge cameras 40 throughout the store location 18, allowing the edge cameras 40 to exchange images or data from any of the connected edge cameras 40 and nodes to designated cameras within the vision mesh network 50. Thus, for example, the adjustable shelving system 10 in the illustrated aisle 12 of Figure 2, with edge cameras 40 installed, can utilize all cameras 40 with FOVs of the shelving units 32, plus the remaining edge cameras 40 installed in the store location 18. Thus, the vision mesh network 50 allows the system to increase the quality and quantity of data input by using images and data from throughout the store location 18 to improve results and accuracy and enhance the decision-making capabilities of the adjustable shelving system 10.

[0019]

[0027] 1-3 are provided for illustrative purposes only, and thus, it will be understood that embodiments are not limited to the configurations shown. For example, there may be more or fewer cameras and aisles than those illustrated, and the aisles may have more or fewer shelf sets, shelving units, shelves, and adjustable shelves, as well as more or fewer features, without departing from the scope of this disclosure.

[0020]

[0028] As illustrated in FIGS. 4A-4C, a customer 70 cannot reach an item on the adjustable shelf 32d because the adjustable shelf 32d is too high. The customer 70 indicates that he or she needs help by pointing or reaching toward the out-of-reach item. The edge cameras 40a, 40b capture and process the behavior of the customer 70, including the customer's gestures, movements, and actions, to determine that the customer 70 is requesting the out-of-reach item on the adjustable shelf 32d. The edge cameras 40 use computer vision, including body recognition techniques, to identify the behavior of the customer 70 as a request for the item. Once the customer request is identified, a designated camera, for example, camera 40a, positions the adjustable shelf 32d at a target location or target height, for example, the target height H shown in FIG. 4C . T1 Send a command to move to

[0021]

[0029] The adjustable shelving system 10 according to some embodiments of the present disclosure includes an adjustable shelf 32d movably connected to a shelving unit 32 by, for example, a bracket 60 connected to an actuator 62 and a motor 64. The shelving unit 32 or the adjustable shelf 32d may also be connected to a power source. A controller 68 is connected to the motor 64 and communicatively connected to the edge camera 40a. In one example, as shown in FIG. 3 , the controller 68 is a node on the mesh network 50. In another example, the controller 68 is not a node on the mesh network 50 but is otherwise connected to the edge camera 40a. For example, the controller 68 may be connected through a wired network, a central server, a control network, or a wireless network, such as a cloud network.

[0022]

[0030] The adjustable shelf 32d is programmably controlled and remotely operated by the controller 68 and the camera 40a. The adjustable shelf 32d may be a "smart shelf" and may communicate with the edge camera 40a. The controller 68 can then use the edge camera 40a to program the adjustable shelf 32d to move from an initial position to a first target height H T1 The first target height H T1 may be adjustable and customizable based on the personal characteristics of the customer 70. The edge camera 40 detects the height of the customer 70 based on the image processed using computer vision and body recognition features, and calculates a first target height H from the personal height and characteristics of the customer 70. T1 In one example, the edge camera 40 may determine if the customer 70 is disabled and adjust the target height or target location accordingly. In another example, the first target height H T1 may be a preselected height.

[0023]

[0031] When the customer 70 completes the interaction with the movable shelf 32d, the movable shelf 32d returns to the initial position shown in Fig. 4A. The edge camera 40 monitors the customer 70 in real time and determines whether the customer 70 reaches the target height H T1The edge camera 40 determines when the customer 70 has finished interacting with the movable shelf 32d. In one example, the mesh network 50, via the edge camera 40, determines that the customer's interaction is complete when the customer 70 moves away from the vicinity of the shelving unit 32. In another example, the camera 40 uses computer vision, such as facial and body recognition, to determine whether the customer 70 is considering removing an item from the movable shelf 32d or returning the item to the movable shelf 32d. For example, the customer 70 may select an item 80, examine the item's nutritional information, and then return the item 80 to the movable shelf 32d. The edge camera 40 may observe the customer 70 within the FOV 42 examining the item 80 and determine that the customer 70 needs more time to complete their interaction with the movable shelf 32d. In a further example, the vision mesh network 50 may instruct the movable shelf 32d to return to its initial position after a time limit has expired. Once the edge cameras 40a, 40b determine that their interaction with the adjustable shelf 32d is complete, the edge camera 40a sends a command to the controller 68 to return the adjustable shelf 32d to its initial position.

[0024]

[0032] The adjustable shelving system 10 may include a sensor 66 located on, near, or around the adjustable shelf 32d to detect an obstruction in the path 38 of the adjustable shelf 32d, for example, to detect whether another customer or item is blocking the way. The sensor 66 may include an infrared light. A step motor may be provided to prevent the adjustable shelf from moving when an obstruction is in the path 38. The adjustable shelving system 10 may also include a light, alarm, or speaker 63 for generating a signal, including light or sound, to alert nearby customers that the adjustable shelf 32d is moving or about to move.

[0025]

[0033] The vision mesh network 50 can monitor customer behavior, actions, and characteristics for safety requirements or preferences. For example, the vision mesh network 50 can recognize a customer as a child and ignore requests from the child to prevent the child from accessing candy, medicine, tobacco, or alcohol. In another example, the vision mesh network 50 can identify a group of shoppers as a couple or a family and honor requests from the child when family members or group members are nearby. The vision mesh network 50 can receive information from loyalty card holders indicating customer preferences regarding interactions with the adjustable shelving system 10 and the vision mesh network 50.

[0026]

[0034] The adjustable shelving system 10 uses data and images acquired, collected, and processed by the mesh network 50 to facilitate customer requests for out-of-reach items on the adjustable shelves 32d. The vision mesh network 50 allows the designated camera 40a to communicate directly with other edge cameras and nodes on the vision mesh network 50. The designated edge camera 40a for a shelving unit 32 can utilize additional images from edge camera 40b and other edge cameras 40 as additional inputs to augment and enhance decisions and results made regarding the customer 70, the out-of-reach item request, and the adjustable shelves 32d. Direct communication increases the speed of data exchange and ensures that potentially multiple data inputs can be processed by the designated edge camera 40a in real time or near time without increasing network traffic, data security requirements, or latency. As used herein, "real time or near time" includes the actual amount of time the customer spent at the adjustable shelves 32d plus a reasonable latency, which may be on the order of a few minutes. By utilizing edge cameras 40 as a mesh of data inputs, the system can increase accuracy without increasing the number and cost of system assets. The vision mesh network 50 broadens the use of available edge cameras 40 for increased effectiveness.

[0027]

[0035] For example, edge camera 40 may observe the behavior of customer 70 while shopping at store location 18, such that edge camera 40 learns that customer 70 likes to read nutritional labels before placing items in their shopping cart. This behavior may be associated with customer 70 as they walk down aisles 12, 14, and 16. Thus, by the time customer 70 arrives at shelving unit 32, mesh network 50 already contains information about the customer's shopping behavior that can be used in decisions regarding adjustable shelving system 10. In this example, the predetermined time limit for adjustable shelf 32d may be extended, or the customer's movement away from adjustable shelf 32d may not be recognized as the end of the interaction.

[0028]

[0036] In some embodiments of the present disclosure, the cameras are visual cameras, as opposed to edge cameras, and no processing is performed there. The visual cameras are connected to a processor that is located at the store location or off-site. The processor performs the necessary computing and sends the necessary instructions to the adjustable shelving. The visual cameras may be connected to each other or to the processor via a network, such as a fiber optic network.

[0029]

[0037] 5A-5C illustrate a customer 72 in an electric shopping cart 74 that advantageously uses the adjustable shelf system 10 according to some embodiments of the present disclosure. The customer 72 indicates a need for assistance by pointing, reaching, nodding, or making another visually or audibly identifiable gesture toward an out-of-reach item or adjustable shelf 32d. The edge camera 40 captures and processes the customer's 72 behavior, including the customer's movements, gestures, and actions, to determine that the customer 72 is requesting an out-of-reach item on the adjustable shelf 32d. The edge camera 40 may use computer vision, including body and facial recognition techniques, to identify the customer's 72 behavior as a request for an item. Once a determination is made, the designated camera 40a directs the adjustable shelf 32d from its initial position to the target location or target height H shown in FIG. 5C. T2 When accommodating a customer 72 in an electric shopping cart 74, the adjustable shelving system 10 sends a command to move to a second target height H at or near the height of the customer 72 seated in the electric shopping cart 74, e.g., adjacent the basket 73. T2 The movable shelf 32d can be lowered at the second target height H, thereby allowing the customer 72 to easily access the products on the lowered movable shelf 32d. T2 may be adjustable and customizable based on the individual characteristics of the customer 72, or may be set to a target height H T2 may be a preselected height. The second target height H T2 is the first target height H shown in FIGS. 4A to 4C. T1 The edge camera 40 monitors the customer 72 in real time and detects whether the customer 72 is at a target height H T2 When the edge camera 40 determines that the interaction is complete, the edge camera 40a instructs the adjustable shelf 32d to return to its initial position via the controller 68, the motor 64, and the actuator 62.

[0030]

[0038] 6A and 6B show multiple customers 70, 72, 76 using the adjustable shelving system 10 according to some embodiments of the present disclosure. Customer 70 is of average height, customer 72 is sitting in an electric shopping cart 74, and customer 76 is a child and shorter than the average adult. As multiple customers 70, 72, 76 arrive at the shelving unit 32, the adjustable shelving system can function in a variety of ways to assist each customer.

[0031]

[0039] In one example, the mesh network identifies which customer 70, 72, 76 arrives first and processes them on a first-come, first-served basis, as described above with respect to FIGS. 4A-5C. The first customer 70 is identified by the vision mesh network as the "control customer" and is the first customer to be assisted. The first customer 70 requests help, for example, by pointing or gesturing toward an out-of-reach item or adjustable shelf 32d. The edge camera 40 uses computer vision, including body and facial recognition techniques, to identify the customer 70's behavior as a request for an out-of-reach item. The designated camera 40a positions the adjustable shelf 32d from its initial position to a first target height H shown in FIG. 6B. T1 The customer 70 sends a command to move to the first target height H T1 The customer 70 completes the transaction by selecting an item from the movable shelf 32d at the first target height H. The edge camera 40 monitors the customer 70 in real time and uses computer vision and body recognition to determine whether the customer 70 reaches the first target height H. T1 When the vision mesh network 50 determines that the "controlling" customer 70 has completed their interaction with the adjustable shelf 32d, the vision mesh network 50 transfers the "controlling customer" designation to the next customer, customer 72.

[0032]

[0040] In some embodiments, the edge camera 40a may command the adjustable shelf 32d to return to its initial position before assisting additional customers 72, 76. In some embodiments, as shown in FIG. 6B , rather than the edge camera 40a returning the adjustable shelf 32d to its initial position for each customer 70, 72, 76, the edge camera 40a may command the adjustable shelf 32d to reach a second target height H T2 to lower, thereby allowing customer 72 in electric car 74 to access adjustable shelf 32d. Once vision mesh network 50 determines that interaction with customer 72 is complete, mesh network 50 designates customer 76 as a "controlling customer." Mesh network 50 then lowers adjustable shelf 32d to a lower, third height H T3 32d to assist the next customer 76, thereby allowing the customer 76 to access the adjustable shelf 32d. Once the vision mesh network 50 determines that the interaction with the customer 76 is complete and that there are no more customer requests pending, the adjustable shelf 32d is commanded to return to its initial position.

[0033]

[0041] In some embodiments, when the vision mesh network 50 receives two or more requests from multiple customers 70, 72, 76, the system 10 identifies a common target height that is accessible to two or more customers and / or most or all of the customers. For example, as shown in FIG. 6B, the adjustable shelf 32d is positioned at a first target height H T1 , the second target height H T2 , and a third target height H T3 first to a third target height H T3 to simultaneously assist customers 70, 72, and 76. In some embodiments, the vision mesh network 50 can recognize when a customer may have a disability, so the lowest target height may be too low and the adjustable shelf 32d may need to be subsequently adjusted to accommodate that particular customer.

[0034]

[0042] As shown in FIG. 7 , the adjustable shelving system 10 includes a shelving unit 132 including adjustable shelves 132a and 132b, which are the two lowest shelves of the shelving unit 132. The adjustable shelves 132a and 132b extend laterally and outwardly away from the rear wall 133 of the shelving unit 132, as indicated by the arrows in FIG. 7 . Items 82 located near the rear wall 133 of the shelves 132a and 132b may be difficult for customers to reach. In this embodiment, the customer 72 indicates a need for help by pointing, nodding, or making another visually identifiable gesture toward the out-of-reach item 82 on the adjustable shelf 132a. The edge camera 40 acquires, captures, and processes the behavior of the customer 72 using computer vision, including body recognition and facial recognition techniques, and identifies the behavior as a request for the out-of-reach item 82. Once a customer request is identified, the designated camera 40a moves the adjustable shelf 132a from its initial position to the target location L shown in FIG. 7 . T1 , thereby making merchandise at the rear of the adjustable shelf 132a more accessible to the customer 72. Additionally, the adjustable shelves 132a, 132b may be moved in any direction. Thus, the adjustable shelf 132d may be commanded to extend both laterally and vertically, thereby allowing the customer 72 riding in the electric shopping cart 74 to easily reach the merchandise 82 without having to lean over or with less effort. Once the vision mesh network 50 determines that the interaction is complete, the vision mesh network 50 commands the adjustable shelf 132a to return to its initial position via the edge camera 40a, controller 68, motor 64, actuator 62, and bracket 60, as described above.

[0035]

[0043] It will be understood that the configurations illustrated in FIGS. 1-7 are provided by way of example only, and that embodiments of the present disclosure are not limited thereto.

[0036]

[0044] 8 is a flow diagram illustrating an adjustable shelving system 100 according to some embodiments of the present disclosure. Input 110 related to one or more customers 70, 72, 76 and / or adjustable shelves 32d (FIGS. 4A-6B) is input to a processor 150. The input 110 is obtained from an edge camera 40 mounted at a store location 18. The input 110 is processed 150 using computer vision, including body recognition and facial recognition techniques, and an output related to the adjustable shelving system 100 is provided 160.

[0037]

[0045] 9 is a flow diagram illustrating an adjustable shelf system 200 according to some embodiments of the present disclosure. Referring now to FIG. 9 in conjunction with FIG. 4C, input 210 regarding one or more customers and / or adjustable shelves 32d is entered into a processor 215 and processed using computer vision, including body recognition techniques, to determine if a customer request 220 is identified. If a customer request 220 is not identified, the processor 215 continues to process the input until a customer request is identified. The input 210 is obtained from an edge camera 40 within the store location 18.

[0038]

[0046] Once the customer request 220 is identified, the adjustable shelf 32d is moved to the target position, i.e., the target height H T1 , allowing the customer 70 to access the items 80 on the lowered adjustable shelf 32d. Input 230 regarding the customer 70 and the adjustable shelf 32d is continuously processed 232 using computer vision, including body and facial recognition techniques, to determine if the customer 70 has finished interacting with the adjustable shelf system 240. If the customer 70 has not finished, the adjustable shelf continues to move to the target height H until the customer is done or the interaction reaches a time limit 260. T1 and the adjustable shelf 32d is returned to its initial position 270. The time limit may be a default time period programmed into the adjustable shelf system 200. The system 200 may also move the adjustable shelf 32d again if height or position adjustment is required.

[0039]

[0047] When the system 200 detects 240 that a customer request has been completed, the system 200 considers whether another customer request has been made or queued during the first customer interaction 250. For example, the edge camera 40 detects that a second customer needs assistance at the shelving unit 32. If no further requests are outstanding, the movable shelf 32d returns to its initial position 270. If further requests are outstanding 250, the system determines whether the shelf needs to be moved to another position to assist the next customer 252, whether the shelf should remain in the target position for an additional time 254, or whether the shelf should return to its initial position. If the shelf needs to move, a new target position is identified and the shelf is commanded to move 256.

[0040]

[0048] As described above, some embodiments of the present disclosure provide a vision mesh network that allows data from multiple cameras to be shared and relevant data to be processed to provide accurate results for the adjustable shelving system. For example, images provided using the vision mesh network may provide information for determining whether a customer is reaching an item on a shelf. Therefore, some type of data processor is required to process the data provided using the mesh network. As described above, according to one embodiment of the present disclosure, each of the edge cameras 40 processes information, and calculations, including analysis of images and other data, are performed on the edge camera 40.

[0041]

[0049] 10, a data processor 600 will be described that communicates with a body recognition module 690 that receives input from the vision mesh 50 (FIGS. 1 and 2). It will be understood that the data processor may be included in any component of the system without departing from the scope of the present disclosure. For example, the data processor may be present in the adjustable shelving system 10 (FIGS. 1 and 2) or may be centrally located.

[0042]

[0050] As illustrated, FIG. 10 is a block diagram of an example data processing system 600 suitable for use in a system according to an embodiment of the present disclosure. Data processing may occur in any of the devices in the system (or in all of the devices, e.g., in each edge camera 40 (FIGS. 1-3)) without departing from the scope of the present disclosure. The data processing system 600 includes a user interface 644, such as a keyboard, keypad, touchpad, voice activation circuitry, or the like, an I / O data port 646, and memory 636 in communication with a processor 638. The I / O data port 646 may be used to transfer information between the data processing system 600 and another computer system or network. These components may be conventional components, such as those used in many conventional data processing systems, and may be configured to operate as described herein. A computer vision module 690, which may include a body recognition module 692 and a facial recognition module 694, processes input from the mesh network 50 and communicates with the mesh network 50 and the data processing system 600.

[0043]

[0051] The foregoing flow logic and / or methods illustrate the functionality and operation of the various services and applications described herein. If embodied in software, each block may represent a module, segment, or portion of code containing program instructions for implementing the specified logical function(s). The program instructions may be embodied in the form of source code, which includes human-readable statements written in a programming language, or machine code, which includes numerical instructions recognizable by a suitable execution system, such as a processor within a computer system or other system. Machine code may be translated from source code, etc. Other suitable types of code include compiled code, interpreted code, executable code, static code, dynamic code, object-oriented code, visual code, etc. These examples are not intended to be limiting in this context.

[0044]

[0052] When embodied in hardware, each block may represent a circuit or multiple interconnected circuits for implementing the specified logical function(s). The circuitry may include any of a variety of commercially available processors, including, but not limited to, AMD® Athlon®, Duron®, and Opteron® processors, ARM® application, embedded, and secure processors, IBM® and Motorola® DragonBall® and PowerPC® processors, IBM and Sony® Cell processors, Qualcomm® Snapdragon®, Intel® Celeron®, Core(2) Duo®, Core i3, Core i5, Core i7, Itanium®, Pentium®, Xeon®, Atom®, and XScale® processors, and similar processors. Other types of multi-core processors and other multi-processor architectures may also be used as part of the circuit. According to some examples, the circuit may also include an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA), and the module may be implemented as a hardware element of the ASIC or FPGA. Furthermore, embodiments may be provided in the form of a chip, chipset, or package.

[0045]

[0053] While each of the foregoing flow logics and / or methods shows a specific order of execution, it is understood that the order of execution may differ from that depicted. Also, operations shown sequentially in the flowcharts may be performed concurrently or with partial concurrence. Furthermore, in some embodiments, one or more of the operations may be skipped or omitted. Furthermore, any number of counters, state variables, alert semaphores, or messages may be added to the logic flows or methods described herein for purposes such as improving usability, accounting, performance measurement, or providing troubleshooting assistance. It is understood that all such variations are within the scope of the present disclosure. Furthermore, not all operations illustrated in the flow logic or methods may be required for novel implementations.

[0046]

[0054] If any operation or component described herein is implemented in software, any one of several programming languages ​​may be used, such as, for example, C, C++, C#, Objective C, Java, Javascript, Perl, PHP, Visual Basic, Python, Ruby, Delphi, Flash, or other programming languages. A software component is stored in memory and executable by a processor. In this regard, the term "executable" refers to a program file in a form that can ultimately be executed by a processor. Examples of executable programs may include, for example, a compiled program that can be converted into machine code in a format that can be loaded into a random-access portion of memory and executed by a processor; source code expressed in a suitable format, such as object code that can be loaded into a random-access portion of memory and executed by a processor; or source code that can be interpreted by another executable program to generate instructions in a random-access portion of memory to be executed by a processor. An executable program may be stored in any portion or component of memory. In the context of this disclosure, a "computer-readable medium" may be any medium (e.g., memory) capable of containing, storing, or maintaining the logic or applications described herein for use by or in connection with an instruction execution system.

[0047]

[0055] Memory is defined herein as an article of manufacture, including volatile and / or nonvolatile memory, removable and / or non-removable memory, erasable and / or non-erasable memory, and writable and / or rewritable memory. A volatile component is a component that does not retain a data value upon loss of power. A non-volatile component is a component that retains data upon loss of power. Thus, memory may include, for example, random access memory (RAM), read-only memory (ROM), hard disk drives, solid-state drives, USB flash drives, memory cards accessed via memory card readers, floppy disks accessed via associated floppy disk drives, optical disks accessed via optical disk drives, magnetic tapes accessed via appropriate tape drives, and / or other memory components, or a combination of any two or more of these memory components. Furthermore, RAM may include, for example, static random access memory (SRAM), dynamic random access memory (DRAM), or magnetic random access memory (MRAM), and other such devices. The ROM may include, for example, a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or other similar memory device.

[0048]

[0056] The devices described herein may include multiple processors and multiple memories, each operating in parallel processing circuits. In such cases, a local interface, such as a communication bus, may facilitate communication between any two of the multiple processors, between any processor and any memory, or between any two of the memories, etc. The local interface may include additional systems designed to coordinate this communication, including, for example, load balancing. The processors may be electrical or of any other available structure.

[0049]

[0057] It should be emphasized that the above-described embodiments of the present disclosure are merely possible examples of implementations, set forth for a clear understanding of the principles of the present disclosure. It is, of course, not possible to describe every conceivable combination of components and / or methodologies, but one skilled in the art will recognize that many additional combinations and permutations are possible. That is, many variations and modifications can be made to the above-described embodiment(s) without substantially departing from the spirit and principles of the present disclosure. All such modifications and variations are intended to be included within the scope of the present disclosure and protected by the following claims.

[0050]

[0058] This disclosure refers to "customer interaction." As used herein, the term "customer interaction" is used broadly to refer to any part of the process for conducting a transaction or action at a retail location, such as selecting an item for purchase. The specific steps involved may vary depending on the retailer, the customer, the store configuration, and the shelving unit.

[0051]

[0059] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting to other embodiments. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly indicates otherwise. It will be further understood that the terms "comprises," "comprising," "includes," and / or "including," "has," and / or "having," as used herein, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. An element described as "for" performing a function, act, and / or operation may be configured or otherwise structured to do so. As used herein, the term "and / or" includes any and all combinations of one or more of the associated list items and may be abbreviated as " / ."

[0052]

[0060] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the various embodiments described herein belong. Furthermore, it will be understood that terms used herein should be interpreted as having a meaning consistent with the meaning in the context of the present specification and the related art, and should not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0053]

[0061] While the foregoing is directed to aspects of the present disclosure, other and further embodiments of the disclosure may be devised without departing from the basic scope thereof, which scope is determined by the following claims.

Claims

1. 1. A movable shelving system for a retail or grocery store, comprising: The movable shelves in the store location, An actuator for moving the movable shelf; a plurality of cameras at said store location, each having a field of view; a vision mesh network having a plurality of nodes in communication with each other; Equipped with At least one of the plurality of cameras is a node of the plurality of nodes on the vision mesh network; A movable shelf system, wherein the vision mesh network receives and processes information about the movable shelf and sends commands to the movable shelf based on the information.

2. The adjustable shelf system of claim 1 , wherein the vision mesh network processes information using a computer vision model.

3. The adjustable shelf system of claim 2 , wherein the computer vision model includes body recognition or facial recognition.

4. The adjustable shelf system of claim 2 , wherein the information includes an image.

5. The movable shelf system of claim 1 , wherein at least one of the plurality of cameras is an edge camera, the edge camera receiving and processing the information and sending the instructions to the movable shelf.

6. The adjustable shelf system of claim 1 , wherein the information comprises an image.

7. 2. The adjustable shelf system of claim 1, wherein the plurality of cameras are edge cameras, the edge cameras being nodes of the plurality of nodes on the vision mesh network, and the edge cameras communicating with each other.

8. The adjustable shelf system of claim 1 , wherein the plurality of cameras are connected together by wire or wirelessly.

9. 10. The adjustable shelf system of claim 1, wherein the vision mesh network determines when a customer requires assistance with an item on the adjustable shelf.

10. 10. The adjustable shelf system of claim 9, wherein the adjustable shelf is lowered to a target height in response to a customer request.

11. 11. The adjustable shelf system of claim 10, wherein the vision mesh network determines if there is an obstruction in the path of the adjustable shelf.

12. 12. The adjustable shelf system of claim 11, wherein the adjustable shelf does not move or stops moving if an obstruction is detected in the path.

13. The adjustable shelf system of claim 11 , wherein the obstacle is a person or an object.

14. 10. The movable shelf system of claim 9, wherein the vision mesh network determines when a customer request is completed and sends instructions to the movable shelf based on the completed request.

15. 10. The adjustable shelf system of claim 1, further comprising a controller connected to said adjustable shelf, said controller being a node on said vision mesh network.

16. 10. The adjustable shelf system of claim 1, further comprising a control network, the adjustable shelf being connected to the control network via a controller, and the vision mesh network being connected to the control network.

17. The adjustable shelf system of claim 16 , wherein the control network is a cloud network.

18. acquiring information about a movable shelf using a first camera associated with the movable shelf at a store location; processing the information over a vision mesh network to obtain a result for the movable shelf, wherein the vision mesh network has a plurality of nodes and the first camera is a node on the vision mesh network; communicating instructions to the movable shelf based on the results via the vision mesh network; A method comprising:

19. The method of claim 18 , wherein processing the information includes using a computer vision model.

20. The method of claim 19 , wherein the computer vision model includes using body recognition or facial recognition.

21. 20. The method of claim 18, wherein the first camera is a first edge camera, the first edge camera being communicatively connected to the movable shelf and to at least one other node on the vision mesh network.

22. 22. The method of claim 21, further comprising a controller connected to the movable shelf, the controller being a node on the vision mesh network and in communication with the first edge camera.

23. The method of claim 22 , wherein the first edge camera processes the information and communicates the instructions to the controller.

24. The method of claim 21 , wherein the first edge camera processes the information and communicates the instructions.

25. When executed by one or more processors, the one or more processors are acquiring information about the movable shelf using a first edge camera associated with the movable shelf; processing the information to obtain a result for the adjustable shelf; communicating instructions to the movable shelf based on the results; and A non-transitory computer-readable medium having stored thereon computer-executable instructions for causing a

Images