Systems and methods for locating items in an intralogistics environment

Abstract

A system and method for mapping and locating items in an intralogistics environment, such as a chaotic warehouse, includes a camera system, a computer system connected to the camera system, and multiple AR markers located on items and / or storage locations. The computer system can search for the AR markers in an image from the camera system and estimate the pose of at least one identified AR marker relative to the camera. The AR markers include at least one ID section and a detection feature. The computer system can further calculate the position and / or orientation of the at least one identified AR marker or the position and / or orientation of the camera, read the at least one ID section to calculate an object ID, and enter the position and / or orientation of the at least one identified AR marker and the object ID into a marker database. Furthermore, a light pointer is mounted together with the camera on a pan-tilt head configured to change the orientation of the light pointer in at least two axes, and the light pointer is configured to point to a selected item.

Description

[Technical Field]

[0001] The present invention relates to a system and method for locating items in an intralogistics environment, such as a chaotic warehouse, and for managing the chaotic warehouse. The system is based on optical patterns recorded by at least one camera.

[0002] Background technology Modern intralogistics environments, such as chaotic warehouses, offer several advantages over fixed-location management, such as better utilization of storage space and simplified storage of new items. However, the complex task of managing warehouse inventory necessitates a warehouse management system. Warehouse management becomes even more complex when goods and storage spaces vary in size and shape.

[0003] A system and method for registering warehouse inventory is disclosed in EP 2 668 118 A1. A robotic device suspended from a rope includes an RFID reader that moves in front of a rack storage area, enabling the RFID reader to scan RFID transponders on items. This system requires that all items have RFID transponders and requires a relatively slow scanning process. Furthermore, the items must be kept in a rack storage area with a relatively flat front.

[0004] Summary of the Invention The problem to be solved by the present invention is to provide a system and method for locating and managing items in an intralogistics environment, which may include a chaotic warehouse. Essentially, embodiments can be used in any type of intralogistics environment, including some types of warehouses, organized warehouses, storage spaces of any size, manufacturing floors, office spaces, etc. The system may be capable of operating without RFID transponders or any other active technology that must be attached to items or warehouse sections. The system may not require racks in warehouse storage areas. Therefore, it should be capable of locating items in any type of structured and unstructured storage space, such as compartments, trays, or parking spaces. The term "item" also refers to storage locations for holding and storing items or goods (including gases and liquids), boxes, pallets, storage boxes, containers, shelves, areas, etc., where items may be located or housed.

[0005] The solution to the problem is set out in the independent claims. The dependent claims relate to further refinements of the invention.

[0006] In one embodiment, a system for locating items in an intralogistics environment, e.g., a chaotic warehouse, and managing such an intralogistics environment, e.g., a chaotic warehouse, includes at least one camera system connected to a computer system. The camera system includes at least one camera, but may include multiple cameras. The computer system includes at least one computer, which may be located on a camera, a cart, a warehouse, or may be cloud-based. The at least one camera is configured to take images and transmit them to the computer system. The images may be transmitted upon request by the computer system, upon user request, or at predetermined intervals after the camera detects movement or a change in the image. Individual images may be transmitted, or there may be a sequence of images, such as a video stream.

[0007] Each of the managed items and / or storage spaces for the items and / or bins for the items (e.g., for bulk goods such as screws) can include at least one optical marker. Such markers include at least one of a barcode, a QR code, or an AR marker, such as an ARUCO marker. This can also be any type of fiducial marker. Such markers can be attached, for example, to a label or sticker, or printed, engraved, e.g., laser engraved, or embedded directly on the surface of the item. Other examples of such AR markers are ARToolKit or ARTag. An AR marker (augmented reality marker) is an image or small object that can be used to align and position an AR object. Here, AR stands for Augmented Reality. A specific AR marker type is the ARUCO marker developed by the Ava group at the University of Cordoba (Spain). Here, the term AR marker includes any of the marker types mentioned herein that may be suitable for the methods disclosed herein. There are no restrictions on the placement of the AR marker. AR markers can be found everywhere, from fixed locations in infrastructure to merchandise and objects.

[0008] The AR marker may include a structure for holding an object ID. Such an object ID may be an article identifier, a part number, or any other suitable code. The code may include bars or rectangular or square fields that may be colored or simply black and white. This structure is referred to herein as an ID section. Additionally, the AR marker may have features, such as a black border or any other structure or pattern configured to be detected. This is referred to herein as a detection feature.

[0009] A computer system can receive images captured by the camera. A detection algorithm in the computer system can search for corners and edges in the image. When the detection algorithm detects distinct corner-edge combinations, a marker can be identified and located in the image. Based on further triangulation, camera parameters, and the real-world size of the marker, the relative position and orientation between the camera and the marker can be calculated. Real-time visual odometry calculations can follow to update the location of the item and the user's position.

[0010] While a wide variety of markers can be used, AR markers offer several advantages. AR markers allow detection from greater distances than most reference markers. As a result, AR markers are detected in more images than others. This aids in later calculation of their location using photogrammetry / SLAM-based algorithms. While the outer contour of a typical barcode or QR code is at least partially determined by the encoding bit pattern and therefore varies from code to code, AR markers offer significant predefined features that can enable a good initial estimate of their pose or angle and / or distance to the camera. While AR markers are specifically mentioned herein, the methods and devices can operate with any type of suitable reference marker, which may include at least information about the marker, e.g., a unique identifier, and some means of recognizability, e.g., a distinct geometric shape.

[0011] The use of markers may be only the first step in evolving the system into a self-learning warehouse solution. Currently, markers solve the computationally heavy problem of quickly identifying unique corner-edge patterns to demonstrate the fundamental ability to learn, manage, and update all items in any warehouse.

[0012] In one embodiment, the computer system is configured to analyze the image and / or search for the AR marker in the image. Further, the computer system may be configured to calculate and / or estimate at least one of a position, an orientation, or a pose of the identified AR marker relative to at least one camera that captured the image. Optionally, the pose and / or the position of at least one of the cameras may be estimated.

[0013] The pose includes the object's position and orientation. The computer system may be further configured to identify the position of at least one camera that captured the image. The computer system may be further configured to calculate the position and / or orientation of at least one identified AR marker. This can be based on the camera's position and the distance and / or angle of the AR code relative to the camera. Alternatively, the position and / or orientation may be calculated by comparing known pairs of 3D and 2D points with geometric shapes in the image. If the image coordinates of each detected code can be assigned to pre-calculated 3D coordinates, it is possible to calculate the camera's position. Theoretically, one marker per image is sufficient to calculate the camera pose. The larger the pixel area of the marker on the image, the more accurate the camera position calculation. If this size is too small, the camera position error will be too large to be useful. The current prototype uses physically very small markers (2 cm x 2 cm), necessitating the use of multiple markers. Nevertheless, in environments with very low marker density, physically larger markers can be used as orientation aids.

[0014] The AR marker pose calculation can be performed using the Perspective-n-Point (PnP) algorithm, which is available as open source. The error increases with distance. Under good conditions, an AR code with a size of 2 cm x 2 cm can be detected at a distance of 2 m.

[0015] A method for locating an item in an intralogistics environment can use a computer system, at least one camera, and a plurality of AR markers assigned to the item. capturing at least one image with at least one camera and transmitting the at least one image to a computer system (120); a computer system searching for an AR marker in at least one image, preferably one of the at least one image; identifying at least two AR markers; reading ID sections of at least two AR markers; determining object IDs of at least two AR markers from the ID section; Optionally, checking the object ID in a marker database to identify at least one known marker; - estimating the relative position of at least one of the at least two AR markers (211) relative to a known marker, preferably on the same image, the remaining (unknown marker); storing at least one estimated location together with the object ID of the AR marker (211) in a marker database (122); may include:

[0016] These steps can be used to initially build a marker database. They can also be used to update the marker database. If there are multiple unknown markers in the image and the database has missing or inaccurate locations, the last two or three steps can be repeated multiple times. This system can be used with absolute or relative locations, or a mixture of both. Essentially, it is sufficient to know only the relative location of a marker relative to other markers. If at least one marker is in a known absolute location, the absolute locations of other markers can be calculated. To improve positioning accuracy and avoid dead spots where it is impossible to find two markers on the same image, multiple markers can exist in known absolute locations within the intralogistics environment.

[0017] The at least one image may be, for example, a composite or stitched image of multiple images taken sequentially or simultaneously by different cameras. The at least one image may include, for example, multiple sub-images taken simultaneously by different cameras having known poses.

[0018] Preferably, the AR markers used for location estimation should be on the same image. To improve the method, multiple image frames taken by the camera can be stitched together to form a larger image, which can show a larger number of markers.

[0019] The method is: a) selecting a point of interest, which may be done from an item database or a marker database or by defining coordinates; b) Follow these steps: capturing at least one image with at least one camera; searching for an AR marker in the image; identifying at least one AR marker; reading an ID section of at least one AR marker; determining an object ID of at least one AR marker from the ID section; looking up the location of at least one AR marker in a marker database; - estimating at least one pose and / or position of the at least one camera based on the position of the at least one AR marker, optionally estimating at least one position of the at least one AR marker by, for example, a SLAM algorithm, and updating a marker database; calculating a route from the camera position to the location of interest; Transferring the route to a user or mobile device Repeat the steps It may further include:

[0020] These steps include a) requesting a target location (point of interest), where, for example, a particular type of item is or should be stored, and b) estimating the actual position of the camera and calculating a route to the point of interest.

[0021] The method includes, after performing the previous steps of sections a) and b) at least once, Displaying a map of the warehouse or a portion thereof showing the location of interest or a route to the location of interest on at least one of the display, the computer screen, and the handheld device. It may further include:

[0022] The method is: performing the displaying step continuously (e.g., while the camera is moving) to interactively guide the user to the location of interest. This may be done while the at least one camera is moving, for example, through an intralogistics environment.

[0023] The method is: moving at least one camera; carrying out the sub-steps of step b) above; and / or Repeat the steps above to display It may further include:

[0024] The method is: capturing at least one image with at least one camera; searching for an AR marker in the image; identifying at least one AR marker; reading an ID section of at least one AR marker; determining an object ID of at least one AR marker from the ID section; looking up the location of at least one AR marker in a marker database; - estimating at least one pose and / or position of the at least one camera based on the position of the at least one AR marker, and optionally estimating at least one position of the at least one AR marker by, for example, a SLAM algorithm, and updating a marker database; Indicating a location of interest relative to the camera, for example, by at least one of a directional indicator (e.g., on a display, monitor, by an LED, or by a key or switch), an augmented reality view, and a light pointer; It may further include:

[0025] The method is: by displaying markers pointing to locations of interest in the virtual or real image; and / or by pointing a pointer, which may include at least one of an optical pointer, a beamer, or a laser scanner, at the location of interest; The step of indicating a location of interest may further be included.

[0026] The method uses an IMU (Inertial Measurement Unit) that is mechanically coupled to the camera. updating the attitude and / or position of at least one of the at least one camera (624) by reading data from the IMU; and / or stabilizing at least one image taken by the camera based on data from the IMU; It may further include:

[0027] At least one of the multiple AR markers may include a detection feature that may include a black border surrounding an ID section, and the ID section may include a bar or a rectangular or square field that may be colored or simply black and white, or any other suitable geometric form.

[0028] A system for locating items in an intralogistics environment may include a computer system and at least one camera. The system may be configured to perform the method according to any one of claims 1 to 12.

[0029] The camera may be handheld by the user, worn on the user's head, attached to AR glasses, or attached to a cart. The camera may also be part of a mobile device such as a mobile phone or tablet PC. There may also be at least one stationary or tiltable camera. If the camera's position is known, the computer system can determine the distance, direction, and orientation of an object within the camera's field of view. Additionally, the movable camera can be used with SLAM (simultaneous localization and mapping) algorithms in the computer system to determine the camera's position and generate and / or update a map of the camera's environment.

[0030] The camera may also be part of a pan-tilt camera head. The pan-tilt camera head may allow the camera's orientation to be changed in at least two axes. The orientation may be stabilized by an IMU (inertial measurement unit) connected to the camera so that the camera points to a selected location in space, which may be an AR code, regardless of camera movement. The camera may be mounted on a gimbal. The IMU may be connected directly to the camera or to a moving device, such as a pan-tilt head, that implements a defined movement of the camera. This can compensate for motion blur and provide a longer exposure time to reduce image noise. The IMU may further enable navigating areas where AR markers are not visible or where the camera moves too quickly for image evaluation, such as across a hallway. Camera stabilization may alternatively / additionally be performed by image processing of the camera image.

[0031] Pan-tilt camera heads can also allow for scanning larger areas of an environment while moving, such as large storage racks across their entire height. Furthermore, specific areas of an environment can be scanned or rescanned while approaching or leaving them. Furthermore, a light pointer, such as a focused LED or laser, can be mounted on the pan-tilt camera head or a separate pan-tilt head. The light pointer can also be a device that does not require pan / tilt, such as a beamer. The light pointer can be configured to point the user to a specific location in the space, such as a specific AR marker or a specific shelf.

[0032] To improve camera localization, additional AR markers can be provided at fixed locations within the warehouse. These can then be immediately used to locate the cameras. Additionally, cameras can have their own AR markers registered in the marker database or assigned virtual AR markers that only have entries in the marker database. The computer system can be configured to identify the location of at least one camera based on at least one of the position values in the marker database, the position values in the camera database, and / or the pose values.

[0033] The AR markers may include reflective materials so that they can be better detected by a user and / or a camera. The AR markers may have different types and / or sizes. There may be larger markers to identify infrastructure that may be distinguishable from a greater distance.

[0034] Taking or initializing an inventory of a warehouse unknown to a computer system is relatively simple. A camera system, or at least one camera of a camera system, can move through the warehouse. At least one marker for each item to be inventoried can be on at least one image of the camera system. Better resolution and better consistency can be obtained using multiple images, which can be taken from different viewpoints and / or different viewing angles. This can include a video stream from a moving camera.

[0035] The computer system evaluates images from the camera system and attempts to identify markers. These markers can be evaluated to identify their object IDs from their ID sections and / or to locate at least one of the object and the camera. Such information can be stored in a marker database, which can include object IDs and associated locations. The marker database can be used, for example, for localization and / or mapping (e.g., SLAM), which can estimate the camera position and / or the location of a new marker with an unknown location based on the known locations of the markers. Alternatively or additionally, further methods, such as triangulation and viewpoint endpoint determination, can be used. The marker database can include further data, such as item-related data such as item identifiers, SKUs (stock keeping units), text descriptions, and available item quantities.

[0036] The database can further be used to guide the user to a selected item that the user can retrieve from the warehouse. The location of the selected item can be shown on a map. The map can further show the shortest route to the item from its actual location. The marker database can also be used by navigation software (also called a navigation app) on a mobile device (also called a handheld device) to navigate the user to the selected item. The user's location and / or orientation can be determined by using a camera and AR marker localization.

[0037] In one embodiment, a laser is provided that can point to a selected item. The laser can be on a cart and driven, pushed, or pulled by a user, or the laser can be stationary. To provide visual guidance to users, conventional systems, such as pick-by-light solutions, require mounting on racks or even directly in each compartment. Some technologies require users to wear devices, such as AR glasses, to guide them. A mobile laser system can be mounted on a cart or in a separate location, covering the entire warehouse to freely guide users while saving technical resources.

[0038] Furthermore, the marker database can be used in AR software or an app on a mobile device, such as a smartphone or tablet PC, to generate an AR camera view. A user can view an image from a camera, which may be part of the mobile device, on a screen on the mobile device. At least one marker can be provided that can indicate the direction to and / or the location of a selected item. A special marking may be applied when the selected item is in direct view. Additional sensors, such as GNSS, accelerometers, or magnetic compasses, found in almost all mobile devices can be used to improve the accuracy, speed, and performance of the location determination.

[0039] In one embodiment, the system is configured to identify special gestures, such as when a user moves a handheld camera near a marker of a selected item, signaling that the item has been found and that additional items should be searched for. For example, there may be additional gestures, such as rotating the camera by, for example, 90 degrees, with a close-up view of the marker, that can indicate that an item has been removed.

[0040] In one embodiment, there may be 3D hand recognition via a single camera that uses artificial intelligence to detect and respond to intuitive human movements such as grasping an item, moving an item from A to B, interacting with the system to change an item list or priority, or canceling a job. This interaction between human and machine may occur via visual, tactile, or audio feedback. This rich addition of features not only improves the intuitive operation of the system, but also supports the well-being of human employees by detecting tremors, heavy labor, or accidents.

[0041] In one embodiment, users and / or carts moving through the warehouse and / or stationary cameras continuously stream images to a computer system, which can then detect the changed locations of individual items. Items in the warehouse can be rearranged simply by moving them from one location to another. Because of the automatic updates, there is no need to register the rearrangement with the computer system.

[0042] In addition to or as a replacement for optical markers, RFID transponders may be provided.

[0043] The system may also be used to manage undesignated storage spaces such as office space, and therefore the system may be used in the very broad field of inventory management in general.

[0044] The system can also be combined with at least one pick-by-write system that can support embodiments in the area of complex picking instructions such as high frequency picking or batch or multi-user picking.

[0045] Putting a new warehouse into service is relatively easy: every item or storage space or bin for an item (e.g., for bulk goods like screws) needs to get an AR marker, preferably on a visible side. The system then begins scanning the warehouse with people or carts moving through the warehouse and builds a marker database of items and their locations.

[0046] The system can also check all items from time to time as they are scanned, allowing for a permanent inventory to be implemented. If an item is missing, the user can be directed to its last known location to retrieve it.

[0047] Because the system operates entirely based on the AR marker, no additional RFID transponders are required, which are more expensive than labels with AR markers. The embodiment minimizes the total amount of hardware and devices. A simple code printed on paper that is always reused is significantly more environmentally friendly than other sensor, button, or display solutions on the market.

[0048] The minimum equipment is a camera and a computer, which are available in almost all smartphones, so that a smartphone equipped with software can perform the steps described herein.

[0049] Although the systems and methods described herein are highly effective in operating in an intralogistics environment, they can also be used in fixed-location warehouses, which can also gain additional benefits such as permanent inventory or recovery of lost or misplaced items.

[0050] In one embodiment, the system is configured for an initial calibration, which can be done with a known pattern, such as a checkerboard. The calibration parameters can be stored, for example, in a camera database.

[0051] In one embodiment, a first camera is provided that is configured to capture a large AR marker, for example, on the floor, and a second camera is provided that is configured to capture a smaller AR marker, for example, in the storage space. The large AR marker may have a size of 16 cm x 16 cm, or may be in a range of 5 cm x 5 cm to 30 cm x 30 cm. The smaller AR marker has a size less than half that of the large AR marker. The larger AR marker on the floor allows for more accurate localization and mapping. In an alternative embodiment, only one camera is provided that can view at least the storage space. The larger AR marker may be on a wall or fixed structure of the storage space.

[0052] In one embodiment, a system for locating an item in an intralogistics environment may include a camera system, a computer system, and may further include a plurality of AR markers on the item and / or storage location. The camera system may have at least one camera that may be configured to take and transmit a plurality of images to the computer system. The computer system may perform the following sequence: searching for an AR marker in the image; identifying at least two AR markers; estimating a relative pose of at least one of the at least two AR markers with respect to each other; Optionally, estimating at least one pose and / or position of the camera. The method may be configured to build and / or maintain a marker database by repeating the steps of:

[0053] The computer system executes the following sequence: calculating a position and optional orientation of at least one AR marker in the marker database based on the chain of relative poses and the known position of at least one AR marker in the marker database, for example by a SLAM algorithm; The method may be further configured to maintain a marker database by repeating the steps of:

[0054] In the method, a camera system including at least one camera captures images from the warehouse and transmits the images to a computer system that evaluates the images and identifies AR markers within the images.

[0055] The computer system then uses at least one of the AR markers to at least one of estimate a distance and / or an angle between the at least one camera and the at least one AR marker and read an object ID of the at least one AR marker.

[0056] If the positions of most of the detected markers in an image are known, the camera pose can be estimated, even if some positions have changed since they were calculated. If larger markers are used, fewer markers or only one marker may be needed. If the AR markers are in fixed positions, the camera position can be estimated. If the AR markers are of an object, the object's position can be estimated. This can be more or less accurate depending on the accuracy of the camera position, which can be accurately known if the camera is fixed and based on the AR markers or estimated by SLAM. The camera position or object position can also be determined by using the pattern of multiple AR markers on the object. Typically, there are few objects that are moved. Therefore, if most of the multiple objects are in a previously recorded spatial relationship, these can also be used as a position reference.

[0057] Based on these position estimates, the position of at least one AR marker is determined by the computer system.

[0058] The location of the at least one AR marker can be stored in the marker database along with the object ID of the at least one AR marker. Further item-related data, such as item identifier, SKU (Stock Keeping Unit), text description, and quantity of available item, may also be stored in the marker database.

[0059] Enterprise resource planning software or warehouse management software initiates requests, usually in the form of work orders containing a list of items to be picked or placed. This information is then passed on and processed by the proposed solution. To pick a specific item from the warehouse, using the item identifier, the computer system retrieves the corresponding item location in the warehouse based on the marker database. This selection can be based on the object ID or any other data in the marker database, such as the SKU.

[0060] Based on the location, a handheld or stationary or movable display (e.g., a display on a cart) can show a map of the warehouse or a portion thereof showing the location of the item and / or showing the shortest route to the item. Additionally, the user may be interactively guided to the location by software running on the handheld device. The software can also show the location of the item or its AR marker on the AR camera view.

[0061] After the user picks the selected item, this can be confirmed by different state-of-the-art techniques. The user can confirm by pressing a sensor button, through voice or gesture detection in the camera view, or by the camera detecting the user picking an item. The camera can also detect the user covering the camera or AR markers. The placement of additional items on the cart can be detected, for example, by weighting or evaluating the camera image of the cart.

[0062] In one embodiment, a cart includes the system described herein. The cart may be further configured to, for example, autonomously move, navigate, and orient itself based on the estimated pose of the camera. Additionally, the cart may have a directional indicator to indicate the direction in which the cart will next move or should next be moved by the operator. The directional indicator may be a screen or touchscreen, a set of illuminated arrows, or any other suitable indicating device, including a sound or voice.

[0063] Embodiments may be able to optimize travel paths and provide a chaotic warehouse with dynamic placement of items, which may enable spontaneous retrieval and spontaneous inventory of items.

[0064] The invention will now be described by way of example, without limiting the general inventive concept, on example embodiments with reference to the drawings, in which: FIG. [Brief explanation of the drawings]

[0065] [Figure 1] FIG. 1 illustrates a system for locating items in an intralogistics environment. [Figure 2] FIG. 1 illustrates an embodiment of an AR marker. [Figure 3] FIG. 1 illustrates a method for locating an item in an intralogistics environment. [Figure 4] FIG. 1 illustrates a method for guiding a user to a selected item in an intralogistics environment. [Figure 5] FIG. [Figure 6] FIG.

[0066] 1 shows a first embodiment of a system 100 for locating items in an intralogistics environment. A camera system 110 is connected to a computer system 120. The computer system 120 may have a marker database 122. A first item or shelf 210 may have first markers 211, 221, 231. A second item 220 may have a second marker 221. A third item 230 may have a third marker 231. The items may be in different locations within the warehouse. The items may have different poses and different orientations.

[0067] The camera system 110 includes at least one camera, shown in this figure, configured to take multiple images of the area of the warehouse where the items are located and transmit these images to a computer system.

[0068] The computer system 120 is configured to identify AR markers 211, 221, and 231 in the image and estimate the pose of the identified AR markers. This pose may be estimated relative to the camera or, preferably, relative to other objects in the image that may have known poses. The pose estimation of any participating object, e.g., a person, camera, or item, is always estimated relative to another participating object. The warehouse floor essentially consists of a 3D relative map of detected markers. Therefore, no prior knowledge needs to be provided to the system. The camera position can be estimated by a SLAM algorithm or any other means for simultaneously determining position and creating a map of the environment. The camera position can also be determined by AR markers on the floor or at other fixed locations. Furthermore, if the camera is mounted in a fixed location, the camera's position can be known. The computer system is further configured to read the ID section of the AR marker. This marker ID can later be assigned information about the product, such as the product ID, name, and quantity, and the assigned information is then stored in the marker database. The marker ID may also contain the product information itself. If the complete pose cannot be estimated, the position of one of the markers may be sufficient.

[0069] The computer system may also be configured to select from a marker database location of the selected item or an AR marker for the item. Based on this, the computer system can display a map of the warehouse or part of the warehouse showing the location of the item. With information obtained from a static or moving camera system, the computer system can directly calculate, create, and update a 3D map of all seen markers representing all known item locations in 3D. This may further display the shortest route to the item from a specific starting point, which may be the user's starting point, or calculate an optimized path for cumulative work orders that need to be batch picked and placed. The map and / or shortest route may be displayed on a handheld device 140, such as a mobile phone or tablet PC, which may already be sufficient to provide all the computing power required by the solution. The map and / or shortest route may also be displayed on a cart that can be used to collect and place the items.

[0070] The handheld device 140 may have software configured to receive at least the position or pose of a selected item or its AR marker from a marker database. Additionally, the handheld device may interactively guide a user to the location of the item. In one embodiment, the handheld device may have augmented reality (AR) software that displays a camera view showing an image of the handheld device's environment and at least one marker that may indicate the direction to the selected item and / or the location of the selected item.

[0071] FIG. 2 illustrates one embodiment of an AR marker. Generally, the AR marker 300 can include a detection means 310 and at least one coding field 320, 330. The AR marker can be any type of suitable marker, such as a QR code, a data matrix, or a 2D code. Furthermore, instead of using an AR marker, products, bottles, merchandise, or any type of infrastructure can be identified, for example, using artificial intelligence. The detection means can be used to recognize the AR marker on an image. The image position can be used to calculate the pose of the AR marker, and the coding field contains a marker ID that can be used as a reference to an object in a database. Preferably, the object ID is a unique ID. Such an object ID can be represented by the pattern of the coding field, which in this example is either black, like coding field 320, or white, like coding field 330. The coding fields can have any other color that makes them distinguishable, and they can have any other shape. Additionally, the AR marker can have any decoration or logo.

[0072] FIG. 3 illustrates a method for locating an item in an intralogistics environment in step 510, in which a camera system including at least one camera captures images from a warehouse and transmits the images to a computer system. In step 511, the computer system evaluates the images and identifies AR markers in the images. In step 512, the computer system can use at least one of the AR markers to calculate a camera pose, for example. If larger markers are used, a single marker may be sufficient. If smaller markers are used, known pairs of 3D and 2D points may be required. For accurate location estimation, a minimum of three, or better still, four, known markers may be required. Furthermore, there may be three different camera poses to provide different views of the AR marker. The known markers may not be on a single line, and the different camera views may not be on a single line. In step 513, the computer system can estimate the location of at least one AR marker.

[0073] Markers can be detected on multiple images to enable accurate 3D reconstruction using photogrammetry, SLAM, or other similar algorithms. At least four markers may be present on each image with detected markers. However, this depends on the size of the markers used. If larger markers are used, one marker per image may be sufficient. All information obtained from all images can be used for the 3D reconstruction of the warehouse. The algorithm can start with a simple triangulation of points and / or markers between two frames to initialize the map. An optimization algorithm can then be used to refine all reconstructed points. Using these known 3D points, more camera poses can be estimated. Images with known camera poses and / or positions can include unknown points in the calculation and triangulate them if they are visible on multiple images. These newly triangulated points are then refined using an optimization algorithm. This step may be repeated until all points and camera poses are calculated. Only points may be saved and reused the next time the system is started. There may be too few markers detected on the images generated to calculate the camera position, so not all points can be reconstructed. Therefore, only images for which the camera position can be calculated can be used for 3D reconstruction. This is just one example of 3D reconstruction. There are several different methods that may require only two markers per image.

[0074] In step 514, the computer system stores the location of at least one AR marker along with the object ID of the at least one AR marker in the marker database. The steps may be performed in the sequence described above. The steps may be automatically repeated multiple times by the system in the background to update the database during productive use of the system. The constantly updated marker locations enable immediate identification, updates, and notification. For example, if a new item is placed in the warehouse without the system's knowledge, the item is recognized and included in the relative warehouse map. A removed item that should be in the warehouse is also detected as missing, and the system can immediately alert the operator. The same method can be used if an item is moved from a first location to a second location in the warehouse without the system's use. When an item is detected in the second location, the system immediately notifies the operator and immediately updates the item's location by removing the first location and updating the second location. When an item is not seen by the camera system, the computer system can notify the operator to search for the lost item and navigate the operator to the location where the item was last seen.

[0075] FIG. 4 illustrates a method for guiding a user to a selected item in an intralogistics environment, e.g., a chaotic warehouse. In step 520, the item is located in the intralogistics environment, as mentioned in previous steps 510-514. These steps may be performed for a longer period, such as hours, days, or weeks, before the next step occurs. In step 521, a computer system selects the location of the selected item from a marker database. Then, at least one of the following steps 522, 523, or 524 is performed. In step 522, a display shows a map of the warehouse or a portion thereof, which indicates the location of the item and can indicate the shortest route to the item. In step 523, software on the handheld device 140 interactively guides the user to the location of the item, and in step 524, AR software guides the user and indicates the location of the item in an AR camera view or with a laser or light pointer. Such an AR camera view can show an image of the handheld device's environment along with at least one marker that can indicate the direction to the selected item and / or the location of the selected item.

[0076] FIG. 5 shows a perspective view of the cart. FIG. 6 shows a front view of the cart. The cart 600 can include a cart frame 610, which can include multiple rollers 612, e.g., four rollers, and an optional handle 614. The cart 600 can be configured to provide a transport space 630 that can hold at least one container 631-636. Feedback buttons 640 and / or indicators assigned to the container or container space may be present. A camera and / or optical pointer head 620 can be mounted on the cart, which may be located on top of the cart to have a free field of view. The camera and / or optical pointer head 620 can include a camera 624 and / or an optical pointer 626. The head can be mounted on a gimbal or similar device that allows the head to be positioned and / or stabilized independently of the movement of the cart. The cart can further hold a computer 650, which can include communication means and further include a battery for powering electronic devices on the cart. [Explanation of symbols]

[0077] 100 Warehouse Management System 110 Camera System 120 Computer Systems 122 Marker Database 140 Handheld Devices 210,220,230 Goods 211,221,231 markers 300 AR marker 310 Location identification means 320,330 Encoding Field 510-514 Steps of a method for locating an item in an intralogistics environment 520-524 Steps of the second method 600 Cart 610 Cart Frame 612 Laura 614 Handle 620 Camera Head 622 Gimbal 624 Camera 626 Light Pointer 630 Transport Space 631~636 Container 640 Button 650 Computer

Claims

1. A method for locating an article in an intralogistics environment, the method using a computer system (120), at least one camera (624), and a plurality of AR markers (211) assigned to articles (210, 220, 230), The steps include capturing at least one image with the at least one camera (624) and transmitting the at least one image to the computer system (120), The computer system (120) performs the steps of searching for an AR marker in at least one image, The steps include identifying at least two AR markers (211), The steps include reading the ID sections of at least two AR markers (211), The steps include determining the object IDs of the at least two AR markers (211) from the ID section, The steps include checking the object ID in the marker database and identifying at least one known marker, A step of estimating the relative position of at least one of the remaining at least one of the at least two AR markers (211) with respect to the known marker, The steps include storing the AR marker (211) along with the object ID and the at least one estimated position in the marker database (122), and Methods that include...

2. a) A step of selecting a location of interest, the selection of which may be performed from an item database or the marker database, or by defining coordinates, b) The following steps: The step of capturing at least one image with the at least one camera (624), A step of searching for the AR marker (211) in the aforementioned image, Steps to identify at least one AR marker (211): The step of reading the ID section of at least one AR marker (211), A step of determining the object ID of the at least one AR marker (211) from the ID section, A step of looking up the position of the at least one AR marker (211) in the marker database, A step of estimating the orientation and / or position of at least one camera (624) based on the position of the at least one AR marker (211), A step of calculating the route from the camera's position to the position of interest, Steps to transfer the aforementioned route to a user or mobile device. The steps of repeating including, The method according to claim 1.

3. After performing the steps described in claim 2 at least once, The step of displaying a map of the warehouse or a portion thereof, showing the location of interest or the route to the location of interest, on at least one of a display, a computer screen, and a handheld device. including, The method according to claim 2.

4. A step of performing the steps described in claim 3 in sequence and interactively guiding the user to the location of interest. including, The method according to claim 3.

5. The steps include moving at least one of the cameras, A step of carrying out the step of section b) described in claim 2, and A step of repeating the steps described in claim 3 and including, The method according to claim 3.

6. The steps include capturing at least one image with the at least one camera (624), The steps include searching for the AR marker (211) in the aforementioned image, The steps include identifying at least one AR marker (211), The steps include reading the ID section of at least one AR marker (211), The steps include determining the object ID of the at least one AR marker (211) from the ID section, The steps include looking up the position of the at least one AR marker (211) in the marker database, A step of estimating the orientation and / or position of at least one camera (624) based on the position of the at least one AR marker (211), A step of indicating the position of interest relative to the camera. including, The method according to claim 1.

7. By displaying a marker indicating the location of interest within a virtual or real image, and / or By directing a pointer, which may include at least one of an optical pointer, a beamer, or a laser scanner, towards the position of interest, The step includes indicating the position of interest, The method according to claim 1.

8. The method uses an IMU (Inertial Measurement Unit) mechanically coupled to the camera, and the method is A step of updating the attitude and / or position of the at least one camera (624) by reading data from the IMU, and / or Steps to stabilize at least one image captured by the camera based on data from the IMU. including, The method according to claim 1.

9. At least one of the plurality of AR markers (211) includes a detection feature which may include a black border surrounding an ID section, and which may include a bar or a rectangular or square field which may be colored or simply black and white. Characterized by, The method according to claim 1.

10. During or immediately after the step of estimating the attitude and / or position of at least one camera (624), A step of estimating the position of at least one of the at least one AR marker (211), or The SLAM algorithm is used to estimate the position of at least one of the at least one AR markers (211), and the marker database is updated. Characterized by, The method according to claim 1.

11. A step of estimating the relative position of at least one of the at least one or two AR markers (211), and Steps to estimate the attitude or position of at least one camera. At least one of them is For example, executed by the SLAM algorithm. Characterized by, The method according to claim 1.

12. The step of indicating the position of interest relative to the camera is performed by at least one of a direction indicator, an augmented reality view, and a light pointer. Characterized by, The method according to claim 1.

13. A system for locating articles (201, 220, 230) in an intralogistics environment, comprising a computer system (120) and at least one camera (624), The system is configured to carry out the method described in any one of claims 1 to 12. system.

14. The aforementioned system, Light pointer, Beema, or Laser scanner Includes a pointer that can contain at least one of the following: The pointer can be mounted on a pan-tilt head (620) configured to change the direction of the pointer on at least two axes. The aforementioned pointer may be stabilized by the IMU, and / or The pointer may be configured to point to a location of interest, and / or The at least one camera (624) may be mounted on a pan-tilt head (620) that can move together with the pointer. Characterized by, The system according to claim 13.

15. The at least one camera (624) is mounted on a specific pan-tilt camera head configured to change the orientation of the camera on at least two axes, the pan-tilt camera head may be stabilized by an IMU, and / or The at least one camera (624) is part of a mobile device such as a mobile phone, PC, tablet PC, VR, or AR glasses, AR goggles, or AR headset. Characterized by, The system according to claim 13.

16. A first camera is configured to image an AR marker at a fixed position, a second camera is configured to image an AR marker on an article, and / or the cart has a direction indicator for indicating the direction in which the cart will next move, and / or The cart is configured to move and / or move autonomously and / or navigate and / or orient itself based on the estimated orientation of the camera. Characterized by, The system according to claim 13.

Images