Automated site inspection based on drone

Abstract

Methods and systems are provided for implementing an automated inventory control mechanism. In an embodiment, an operational workflow providing automated inventory control includes automated data collection and automated image data analysis. Automatic data collection includes capturing image data of a storage facility by a capturing device, such as an unmanned aerial vehicle system (UAS). Automated image data analysis includes functions of detecting objects (e.g., containers, trailers, slots, and / or other objects) appearing in the image data (e.g., using a machine learning (ML) model), identifying objects (e.g., objects in the image data, etc. In some embodiments, the method may include collecting image data (e.g., using an ML model), inspecting the object (e.g., including determining a condition and / or location of the object) based on the collected image data and metadata associated with and related to the image data (e.g., using analysis based on advanced mathematical rules), and / or generating results available for use by an inventory management system.

Description

Technical Field

[0001] The present disclosure relates generally to automated inventory management, and more particularly to systems for automated inventory control based on unmanned aerial systems (UAS). Background Art

[0002] Freight is the lifeblood of trade and is extremely important to our society. Freight is typically made up of individual containers that are transported via a combination of ships, planes, trains, and trucks. While in transit, these containers may be stored in yards or intermodal storage facilities, which support the storage and transportation of containers via different modes of transportation. These intermodal storage facilities are typically arranged in batches, with each facility having multiple rows of containers, each row having multiple slots, and each slot can accommodate a single container. The location of a container in an intermodal storage facility can include batches, rows, and slots, indicating the location of the container in the intermodal storage facility.

[0003] However, accurately maintaining inventory at intermodal storage facilities is very challenging. For example, determining the actual location of containers in an intermodal storage facility is difficult because containers are often not at their designated locations, and current inventory control systems are not robust enough to handle discrepancies between the designated and actual locations of containers. For example, a first container may be assigned to Lot A, Row 3, Location 28, but the user responsible for placing the first container at the appropriate location makes a mistake and places the first container at Lot A, Row 3, Location 27. Due to this user error, the inventory control system may assign the second container to Lot A, Row 3, Location 27 because that location was incorrectly deemed available, and current systems do not have the ability to identify and / or correct this discrepancy. Furthermore, this problem is compounded because when the user responsible for placing the second container at the appropriate location (Lot A, Row 3, Location 27) is actually unavailable, they may place the second container at a different location to avoid delays.

[0004] Some manual solutions implemented to resolve the discrepancy between the designated and actual locations of containers may involve users manually verifying the location of each container in an intermodal storage facility to ensure that the actual location of each container matches the designated location of each container. However, these manual solutions are cumbersome and impractical given the number of containers stored in these intermodal storage facilities. Furthermore, manual verification of container locations is time-consuming and there is a lag between when an error occurs and when it is discovered, thus failing to provide a robust solution to the problems of current inventory control systems. Summary of the invention

[0005] The present disclosure achieves technical advantages as a system, method, and computer-readable storage medium that provide automated inventory control functionality. The systems and techniques of the embodiments provide an improved inventory management system with greater accuracy, reduced lag time, automated collection of data associated with intermodal storage facilities, and automated analysis of collected image data. The present disclosure provides a system that is integrated into a practical application with meaningful limitations because, in some embodiments, the automated collection of image data can utilize an automated data collection system based on an unmanned aerial system (UAS) that is configured to capture image data associated with an intermodal storage facility using an image capture device.

[0006] The present disclosure provides a technical solution that is lacking in traditional systems because, in some embodiments, automatic analysis of collected image data may include the following functions: detecting objects (e.g., containers, trailers, spaces or slots, and / or other objects) appearing in the image data (e.g., using a machine learning (ML) model), identifying objects appearing in the image data (e.g., using an ML model), inspecting the identified objects based on the collected image data and metadata associated with the image capture device (e.g., using advanced mathematical analysis) (e.g., including determining the condition and / or location of the identified objects), and / or generating results that can be used by an inventory management system.

[0007] Thus, the present disclosure discloses concepts that are inseparable from computer technology, such that the present disclosure provides the following technical advantages: at least inventory control, such as, in particular, container identification and inspection (e.g., including determining container location and / or condition), can be provided on demand and in near real time (e.g., sub-second processing), which allows inventory management systems used to manage daily operations to not only be effectively integrated into intermodal storage facilities, but also to more accurately and quickly represent the inventory status and / or location of various objects in the intermodal storage facilities compared to current manual solutions.

[0008] In a particular embodiment, an automated inventory control system is provided. The system includes at least one processor, and a memory operably coupled to the at least one processor and storing processor-readable code, which, when executed by the at least one processor, is configured to perform operations. The operations include acquiring a plurality of input images, detecting at least one object in at least one of the plurality of input images, and associating metadata associated with at least one frame with at least one of the plurality of input images. In an embodiment, at least one frame corresponds to at least one image. The operations also include determining an identity of the at least one object detected in the at least one image based at least on the at least one image, determining a location of the at least one object within a storage facility based at least in part on the metadata associated with the at least one image, and outputting one or more of the identity of the at least one object and the location of the at least one object.

[0009] In another embodiment, an automated inventory control method is provided. The method includes acquiring a plurality of input images, detecting at least one object in at least one of the plurality of input images, and associating metadata associated with at least one frame with at least one of the plurality of input images. In an embodiment, the at least one frame corresponds to at least one image. The method also includes determining an identity of at least one object detected in the at least one image based at least on the at least one image, determining a location of the at least one object within a storage facility based at least in part on the metadata associated with the at least one image, and outputting one or more of the identity of the at least one object and the location of the at least one object.

[0010] In another embodiment, a computer-based automated inventory control tool is provided. The computer-based tool may include a non-transitory computer-readable medium having computer code stored thereon, which, when executed by a processor, causes a computing device to perform operations. The operations include acquiring a plurality of input images, detecting at least one object in at least one of the plurality of input images, and associating metadata associated with at least one frame with at least one of the plurality of input images. In an embodiment, at least one frame corresponds to at least one image. The operations also include determining an identity of the at least one object detected in the at least one image based at least on the at least one image, determining a location of the at least one object within a storage facility based at least in part on the metadata associated with the at least one image, and outputting one or more of the identity of the at least one object and the location of the at least one object.

[0011] The above is a relatively broad overview of the features and technical advantages of the present disclosure in order to facilitate a better understanding of the following detailed description. Additional features and advantages of the present disclosure will be described below, which constitute the subject matter of the claims. It should be understood by those skilled in the art that the disclosed concepts and specific embodiments can be easily used as a basis for modifying or designing other structures to achieve the same purpose of the present disclosure. It should also be recognized by those skilled in the art that such equivalent constructions do not depart from the spirit and scope of the present disclosure as described in the appended claims. The novel features of the organization and method of operation that are considered to be characteristic of the present disclosure, as well as further objects and advantages, can be better understood through the following description in conjunction with the accompanying drawings. However, it should be clearly understood that each figure is used for illustration and description purposes only and does not represent a definition of the scope of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] For a more comprehensive understanding of the present disclosure, reference may be made to the following description in conjunction with the accompanying drawings.

[0013] Figure 1 is a basic block diagram of an example system configured to provide automated inventory control capabilities and functionality consistent with embodiments of the present disclosure.

[0014] Figure 2 is a block diagram illustrating an example of the operation of an automated inventory control system implemented according to an embodiment of the present disclosure;

[0015] Figure 3 An example of object detection according to an embodiment of the present disclosure is shown; and

[0016] Figure 4 A high-level flow diagram 400 is shown of system operation according to an embodiment of the present disclosure.

[0017] Figure 5 An example algorithm for determining the viewing direction of an object in an image according to an embodiment of the present disclosure is shown.

[0018] It should be understood that the drawings are not necessarily drawn to scale and that the disclosed embodiments are sometimes shown in diagrammatic form and in partial view. In some cases, details that are not necessary for understanding the disclosed methods and apparatus or that render other details difficult to understand may have been omitted. Of course, it should be understood that the present disclosure is not limited to the specific embodiments shown herein. DETAILED DESCRIPTION

[0019] The disclosure and its various features and advantageous details set forth in the following written description will be more fully explained with reference to the non-limiting examples included in the accompanying drawings and the contents detailed in the specification. Descriptions of well-known components are omitted herein so as not to unnecessarily obscure the main features described herein. The following examples are intended to facilitate understanding of the implementation and practice of the disclosure. One of ordinary skill in the art will understand the disclosure as meaning that any suitable combination of the following functions or exemplary embodiments can be combined to achieve the claimed subject matter. The disclosure includes the number of representative species within the genus, or structural features common to members of the genus, so that one of ordinary skill in the art can identify members of the genus. Therefore, these examples should not be construed as limiting the scope of the claims.

[0020] One of ordinary skill in the art will understand that any system claim set forth herein encompasses all elements and limitations disclosed therein, and therefore requires that each system claim be considered as a whole. Any reasonably foreseeable item having a functional relationship with a claim also falls within the relevant scope. The Examiner has searched prior art in patents and other published literature after having a full understanding of the invention disclosed and claimed in the non-provisional application. Therefore, as evidenced by the issuance of this patent, the prior art fails to disclose or teach the elements and limitations set forth in the claims supported by the specification and drawings, and therefore the claims set forth are patentable under the applicable rules and laws of this jurisdiction.

[0021] Various embodiments of the present disclosure are directed to systems and techniques that provide automated inventory control functionality. The systems and techniques of the embodiments provide an improved inventory management system with more accurate, less lag time, automated collection of data associated with an intermodal storage facility, and automated analysis of collected image data. In embodiments, the automated collection of image data may utilize an automated data collection system based on an unmanned aerial system (UAS), which is configured to capture image data associated with an intermodal storage facility using an image capture device. In embodiments, the automated analysis of the collected image data may include the following functions: detecting objects (e.g., containers, trailers, spaces or slots, and / or other objects) that appear in the image data (e.g., using a machine learning (ML) model), identifying objects that appear in the image data (e.g., using an ML model), inspecting the identified objects based on the collected image data and metadata associated with the image capture device (e.g., using advanced mathematical analysis) (e.g., including determining the condition and / or location of the identified objects), and / or generating results that can be used by the inventory management system. Among the several benefits of the technical improvements provided by the features of the disclosed systems and methods is that inventory control, such as, in particular, container identification and inspection (e.g., including determining container location and / or condition), can be provided on demand and in near real time, which allows inventory management systems used to manage daily operations to not only be effectively integrated into intermodal storage facilities, but also to more accurately and quickly represent the inventory status and / or location of various objects in the intermodal storage facility than current manual solutions.

[0022] In some embodiments, the application of inventory control may involve objects (e.g., containers, trailers, spaces or slots, and / or other objects) in an intermodal storage facility. However, it should be appreciated that the technology disclosed herein may also be applicable to other applications of inventory control and / or management. For example, the technology disclosed herein may also be applicable to a warehouse to manage retail inventory, or may be applicable to managing vehicle inventory in a vehicle parking lot. Therefore, the discussion herein with respect to containers in an intermodal storage facility should not be construed as limiting in any way.

[0023] Figure 1 is a block diagram of an exemplary system 100 configured with capabilities and functions to provide automated inventory control according to an embodiment of the present disclosure. Figure 1As shown, the system 100 may include a server 110, a user terminal 140, a capture device 130, and a network 145. These components and their individual components may operate in conjunction to provide functionality as discussed herein. For example, in operation according to an embodiment, the user terminal 140 may facilitate a user to interact with the system 100, such as through the network 145, to initiate an operational workflow to automatically collect image data, identify an object (e.g., a container, a trailer, a free space or slot, and / or other object), inspect an object, determine the location of an object, and / or verify or confirm the expected location of an object in an intermodal storage facility. The operational workflow may be initiated by a user scheduling and / or executing a flight plan (e.g., specifying one or more parameters that the capture device 130 may use to capture image data), which flight plan may include a flight path (e.g., including a mode and / or range) for the capture device 130 to follow, or may be pre-scheduled to be executed at a specific date / time or specific periodic interval. While executing the flight plan, the capture device 130 may follow the flight path while capturing image data of the intermodal storage facility according to the one or more parameters. The image data may include image data of objects in the intermodal storage facility. The captured image data may be uploaded or fed to the server 110 via the network 145 for processing. In an embodiment, the image data may include one or more videos of an object. The various components of the server 110 may operate in concert to segment the one or more videos into separate frames, associate metadata of the capture device 130 with the separate frames, and identify frames including at least one object (e.g., using an ML model). As used herein, a frame including at least one object may refer to a frame in which at least one object (e.g., a container, a tail, an empty space or a vacancy, and / or another object) appears. Identifying a frame including at least one object may also be referred to as detecting at least one object in a frame. The various components of the server 110 may operate in concert to process (e.g., using an ML model) a frame including at least one object to identify at least one object detected in the frame. For example, the type of object may be determined, such as a container, a trailer, an empty space or a slot, another object, and the like. In an embodiment, when the object includes at least one container, an identification (ID) of the at least one container may be determined. The various components of the server 110 may operate in concert to examine at least one object appearing in a frame. In embodiments, inspecting the at least one object may include determining (e.g., using advanced mathematical analysis) a location of the at least one object in the intermodal storage facility based at least in part on image data and metadata associated with a frame in which the at least one object appears. In embodiments, the location of the at least one object may refer to global positioning system (GPS) coordinates, geographic information system (GIS) coordinates (e.g., GIS coordinates that may include lot indicators, row indicators, and / or slot indicators), and / or any other physical location measurement.In some embodiments, inspecting at least one object may include determining a condition of the at least one object based at least in part on image data of a frame in which the at least one object appears. In an embodiment, the condition of the at least one object may refer to a physical condition of the at least one object and may include any damage to the object, whether the container is open or closed when the object is a container, whether the object is properly placed, whether the slot is empty when the object is a slot, etc. In some embodiments, the various components of the server 110 may operate in conjunction to generate and output results that may include the identification, location, and / or condition of the at least one object. In an embodiment, the results may be output to the user terminal 140 for presentation to a user or may be output to an external system.

[0024] It is worth noting that in some embodiments, the central controller manages the operational workflow described herein. The central controller can be used to manage queue services including messages between different components of the server 110. In these embodiments, the functions of the various components of the server 110 can be controlled through queue messages, which can be configured to carry information and data between the various components and cause the various components to perform operations according to the functions described herein.

[0025] The following is Figure 1 A more detailed discussion of the functional modules of the system 100 is provided. The discussion of the system 100 will be further based on Figure 2 The example shown is carried out. Figure 2 1 is a block diagram showing an example of the operation of an automatic inventory control system implemented according to an embodiment of the present disclosure. It should be noted that the functional modules and components of the system 100 in the embodiment of the present disclosure can be implemented using a processor, an electronic device, a hardware device, an electronic component, a logic circuit, a memory, a software code, a firmware code, etc. or any combination thereof. For example, one or more functional modules, or parts thereof, can be implemented as discrete gate or transistor logic, discrete hardware components, or a combination thereof, configured to provide logic for performing the functions described herein. Additionally or alternatively, when implemented in software, one or more functional modules or parts thereof may include code segments running on a processor to provide logic for performing the functions described herein.

[0026] It is also noted that the various components of system 100 are shown as single and independent components. However, it is understood that each of the various illustrated components may be implemented as a single component (e.g., a single application, server module, etc.), may be functional components of a single component, or the functionality of these various components may be distributed across multiple devices / components. In such embodiments, the functionality of each respective component may be aggregated from the functionality of multiple modules located in a single or multiple devices.

[0027] It should also be noted that the functions of the different functional modules in the system 100 described herein are provided for illustrative purposes and not as a limitation, and the functions provided by the different functional modules may be combined into one component, or may be provided by computing resources in a cloud environment accessible via a network, such as one of the networks 140 .

[0028] The user terminal 140 may be implemented as a mobile device, a smart phone, a tablet computing device, a personal computing device, a laptop computing device, a desktop computing device, a computer system of a vehicle, a personal digital assistant (PDA), a smart watch, other types of wired and / or wireless computing devices, or any portion thereof. The user terminal 140 may be configured to provide a GUI structured to facilitate user interaction with the system 100 (e.g., via the network 140) to execute and utilize the functions provided by the server 110, and may allow the user to initiate an operational workflow including automatic collection of image data and automatic analysis of image data. In an embodiment, the automatic image data collection may include executing a flight plan, followed by the capture device 130 capturing image data associated with the intermodal storage facility. The automatic analysis of the image data may include identifying objects (e.g., containers, trailers, empty slots, and / or other objects) in the image data, inspecting the objects based on the image data, determining the location of the objects, and / or verifying or confirming the expected location of the objects in the intermodal storage facility. For example, in some embodiments, a user may specify one or more parameters for an automated flight plan that may include a flight path (e.g., including a mode, range, etc.) to be followed by a capture device 130 within the intermodal storage facility to capture image data of objects stored in the intermodal storage facility. In some embodiments, a user may manually specify a flight path that a capture device 130 may follow during an automated flight. In some embodiments, a user may specify an area to be inspected (e.g., by drawing or placing a box of a particular size and / or shape, such as Figure 22, or by specifying a location to be inspected (e.g., Lot A, rows 1 and 2), the system may automatically determine a flight path (e.g., determine an optimized automated flight path) to be followed by the capture device 130 based on one or more parameters specified by the user. The one or more parameters may include: a flight altitude at which the capture device 130 is to fly when capturing image data, a flight speed indicating a speed at which the capture device 130 is to fly when capturing image data, a capture quality indicating a quality of image data to be captured by the capture device 130, a focus altitude indicating an altitude at which the capture device is to focus to capture image data, a type of image data to be collected, a format of image data to be collected, etc. In this manner, a user may specify automated data collection as part of an operational workflow that may be automatically executed (e.g., without the need for a traditional pilot or observer) and may be repeatable, as automated flight plans may be stored for later execution or may be scheduled for execution at regular intervals.

[0029] In an embodiment, the user terminal 140 may allow the user to execute the flight plan by activating a control element of the GIU provided by the user terminal 140. In an alternative or additional embodiment, the flight plan may have been previously scheduled for execution at a specific date / time or at a specific periodic interval, and in this case, the flight plan may be automatically executed without user activation. As will be explained in further detail below, when executing the flight plan, the capture device 130 may follow the flight route while capturing image data according to one or more parameters.

[0030] In some embodiments, user terminal 140 may be configured to present (e.g., display) results output by server 110, which may include identification, location, and / or condition of objects detected in image data captured by capture device 130. In some embodiments, the results are output to an external system, such as an inventory management system that manages and tracks inventory at an intermodal storage facility.

[0031] Capture device 130 may be configured to capture image data of objects in an intermodal storage facility. For example, in an embodiment, capture device 130 may be or may include an image capture device (e.g., a camera) configured to capture or take still pictures or videos. In some embodiments, capture device 130 may be a vehicle with a camera mounted on it. For example, the vehicle may be a ground vehicle (e.g., a car or truck), or may be an aircraft (e.g., a manned aircraft, such as an airborne or remotely piloted aircraft, or UAS). Capture device 130 may be configured to avoid obstacles, and / or may be configured to traverse an intermodal storage facility at a low speed (e.g., a speed not exceeding ten miles per hour) or at a speed that may exceed ten miles per hour.

[0032] It is worth noting that, although the present discussion focuses on UAS as a capture device (e.g., a UAS with a camera mounted on the UAS), the techniques disclosed herein are also applicable to situations where the capture device includes a manned aerial system or a ground vehicle. For example, in some embodiments, a camera may be mounted on a ground vehicle, and the camera may be configured to capture image data of an object as the vehicle travels through an intermodal storage facility in accordance with a flight plan (e.g., following a "flight" plan that may set a ground route for the ground vehicle to follow). Therefore, the discussion of UAS should not be construed as limiting in any way.

[0033] It should also be noted that, as used herein, UAS may refer to unmanned aerial vehicle, and may include autonomous or ground-controlled aircraft, such as drones, multirotors, quadrotors, fixed-wing aircraft, etc.

[0034] The capture device 130 may be configured to traverse the intermodal storage facility in accordance with a flight path (e.g., speed, height of the image capture device, capture quality, etc.) while capturing image data including objects (e.g., containers, trailers, empty slots, and / or other objects) in the intermodal storage facility. Figure 2 As shown, capture device 130 may include capture device 130a, and capture device 130a may be configured to traverse the intermodal storage facility where lot A is located to capture image data according to a flight plan, which flight plan may include capturing image data of lot A, rows 1 and 2.

[0035] As described above, the capture device 130 may be configured such that the capture angle (e.g., tilt angle) of the image capture device (e.g., a camera mounted on the capture device 130) directs the camera toward the intermodal storage facility including the object so as to capture images or video of the object as the capture device 130 flies along the flight route according to the flight plan. For example, in an embodiment where the object includes a container (e.g., Figure 2 ), the capture device 130 may be configured to fly over the multimodal facility (or in some cases, fly outside and adjacent to the multimodal storage facility, such as the capture device 130c), along a path that deviates from the containers in the intermodal storage facility, so that the image capture device of the capture device 130 may have a line of sight to one end of the container (e.g., the end containing the container logo), which may not be the top of the container. For example, Figure 2 As shown, the capture device 130a can fly along a path that deviates from rows 1 and 2 so as to be able to see the end sides of the containers stored in rows 1 and 2. In this way, the captured image data can include images or videos of the containers, including the side ends of the containers, which may include the IDs of the containers.

[0036] In an embodiment, the image data captured by the capture device 130 may include metadata associated with the capture device 130 recorded by the capture device 130 as the capture device 130 traverses the intermodal storage facility. For example, at periodic intervals, the capture device 130 may record or sample metadata, which may include the location of the capture device 130, the orientation of the capture device 130 (e.g., the orientation of the UAS and / or the orientation of the gimbal on which the image capture device (e.g., camera) is mounted), the height of the capture device, the speed of the capture device, a timestamp of the capture device 130 when the metadata is recorded, etc. The sampling rate of the metadata (e.g., the interval at which the metadata is recorded) may be frequent, for example, between 5-15 Hertz (Hz), the sampling rate being equal to the number of frames per second of the captured image data, etc. In another embodiment, the metadata sampling rate may be between 10 Hz and 60 Hz. In another embodiment, the sampling rate may depend on a combination of the sampling rate of the camera and the sampling rate of the GPS. For example, under certain hardware conditions, a sampling rate of 10 Hz may provide sufficient accuracy and speed. In some embodiments, the metadata may be part of a frame-level recording of the video, or may be scaled or interpolated to simulate a frame-level recording. For still images, metadata can be obtained or retrieved from the image's internal file information.

[0037] In embodiments, the capture device 130 may include an onboard memory (not shown) configured to store captured image data, including image data associated with the intermodal storage facility and metadata associated with the capture device 130. In these embodiments, the captured image data may be stored on the onboard memory for subsequent offloading (e.g., to the server 110 via the network 145). In some embodiments, the capture device 130 may be configured to stream the captured image data in real-time (e.g., via the network 145 or another network not shown) to the server 110. In these embodiments, the capture device 130 may transmit, send, or upload the image data to the server 110 in real-time or near real-time as the image data is captured. The real-time streaming functionality of the capture device 130 may be in addition to or in lieu of storing the captured image data on the onboard memory of the capture device 130 for subsequent offloading to the server 110.

[0038] In some embodiments, system 100 may include more than one capture device 130. For example, Figure 2As shown, the system 100 may include capture devices 130a-130c. In these embodiments, a group of capture devices (e.g., multiple capture devices) may be used to capture image data of objects in the intermodal storage facility. In some embodiments, each of the multiple capture devices may operate individually and may collect and upload image data of the object for processing by the server 110. For example, each of the multiple capture devices may be assigned to a different area of ​​the intermodal storage facility, and each of the multiple capture devices may be used to identify and inspect different objects in the intermodal storage facility (e.g., different objects in different areas). In some embodiments, the multiple capture devices may operate in a redundant mode, and in this case, some of the multiple capture devices may be used to repeat the results associated with other capture devices. For example, capture devices 130a, 130b, and 130c may be assigned to an area surrounded by a box 220 of the intermodal storage facility. In this way, capture devices 130a, 130b, and 130c may be operated to capture image data of the same object (i.e., objects in row 1 and row 2). Results associated with image data captured by one capture device may be used to verify, confirm, or otherwise validate results associated with image data captured by another capture device, as the results should be similar. In some embodiments, a redundancy mode may include designating some capture devices as being on standby to replace other capture devices in the event of a problem. For example, capture device 130c may be designated as being on standby. In this case, if a problem occurs with capture devices 130a or 130b (e.g., one of the capture devices fails, runs out of battery, is damaged, or is otherwise inoperable), capture device 130c may be activated to replace the problematic capture device, thereby ensuring continuity of operation.

[0039] In some embodiments, the capture device may be configured to capture image data from multiple rows simultaneously. For example, capture device 130a may be configured to capture image data from row 1 and row 2 simultaneously.

[0040] In an embodiment, the server 110, the capture device 130, and the user terminal 140 may be communicatively coupled via a network 145. The network 145 may include a wired network, a wireless communication network, a cellular network, a cable transmission system, a local area network (LAN), a wireless local area network (WLAN), a metropolitan area network (MAN), a wide area network (WAN), the Internet, a public switched telephone network (PSTN), etc.

[0041] The server 110 may be configured to perform automatic analysis on image data collected as part of an operational workflow. In embodiments, according to embodiments of the present disclosure, the automatic analysis of image data may include ingesting image data, detecting objects appearing in the image data, identifying objects based on the image data, inspecting objects based on the image data, and generating output including automatic analysis results. This functionality of the server 110 may be provided by the coordinated operation of various components of the server 110, as will be described in more detail below. Although Figure 1 A single server 110 is shown, and it should be understood that the server 110 and its various functional modules may be implemented as a single device or distributed across multiple devices having their own processing resources, the aggregate functionality of which may be configured to perform the operations specified in the present disclosure. Figure 1 The components of server 110 are shown as single and independent blocks, and each of the various components of server 110 can be a single component (e.g., a single application, server module, etc.), can be a functional component of the same component, or the functions can be distributed on multiple devices / components. In such an embodiment, the functions of each corresponding component can be aggregated by the functions of multiple modules located in a single or multiple devices. In addition, the specific functions described for a specific component of server 110 may actually be part of a different component of server 110, and therefore, the description of the specific functions described for a specific component of server 110 is only for illustrative purposes and is not limited in any way.

[0042] like Figure 1 As shown, the server 110 includes a processor 111, a memory 112, a database 124, an I / O module 120, an image processor 121, and an object inspector 122. The processor 111 may include a processor, a microprocessor, a controller, a microcontroller, a plurality of microprocessors, an application specific integrated circuit (ASIC), an application specific standard product (ASSP), or any combination thereof, and may be configured to execute instructions to perform operations according to the disclosure herein. In some embodiments, the implementation of the processor 111 may include a code segment (e.g., software, firmware, and / or hardware logic) that may be executed in hardware (e.g., a processor) to perform the tasks and functions described herein. In other embodiments, the processor 111 may be implemented as a combination of hardware and software. The processor 111 may be communicatively coupled to the memory 112.

[0043] The memory 112 may include one or more semiconductor memory devices, read-only memory (ROM) devices, random access memory (RAM) devices, one or more hard disk drives (HDDs), flash memory devices, solid-state drives (SSDs), erasable ROMs (EROMs), compact disk ROMs (CD-ROMs), optical disks, other devices configured to store data in a persistent or non-persistent state, network storage, cloud storage, local storage, or a combination of different memory devices. The memory 112 may include a processor-readable medium configured to store one or more instruction sets (e.g., software, firmware, etc.) that, when executed by a processor (e.g., one or more processors of the processor 111), perform tasks and functions as described herein.

[0044] The memory 112 may also be configured to facilitate storage operations. For example, the memory 112 may include a database 124 for storing various information related to the operation of the system 100. For example, the database 124 may store ML models, mathematical models, rule models, and / or other models that may be used by components of the server 110 to analyze and process image data to detect objects appearing in the image data, identify objects based on the image data, and inspect objects based on the image data according to embodiments of the present disclosure. In an embodiment, the database 124 may provide storage for image data collected by the capture device 130, including image data associated with the intermodal storage facility and metadata associated with the capture device 130. For example, the image data collected or captured by the capture device 130 may be obtained by the server 110 (e.g., via the I / O module 120) and may be stored in the database 124 for further analysis (e.g., by the image processor 121 and the object inspector 122). In an embodiment, database 124 may provide storage configuration data, such as configuration of an automated flight plan for data collection (e.g., periodicity of flight plan execution, one or more parameters, including flight route, flight mode, flight range, flight altitude, flight speed, video capture speed, focus altitude, captured video quality, image data type to be collected, etc.), which may be used in the automated data collection phase of the operational workflow described herein. Database 124 is shown as being integrated into memory 112, but in some embodiments, database 124 may be provided as a separate storage module, or may be provided as a cloud-based storage module. Additionally or alternatively, database 124 may be a single database, or may be a distributed database implemented by multiple database modules.

[0045] The I / O module 120 may be configured to perform input and / or output operations for the server 110 as part of the operational workflow described herein and performed in accordance with the present disclosure. In an embodiment, the input functionality of the I / O module 120 may include an automatic data ingestion module configured to automatically ingest, retrieve, collect, download, or otherwise receive image data collected by the capture device 130, including image data associated with the intermodal storage facility and metadata associated with the capture device 130. As described above, the image data captured by the capture device 130 may be stored in an onboard memory of the capture device 130 for subsequent retrieval, in which case the image data is not provided to the server 110 until after the capture device 130 completes a flight to automatically collect the image data, or the image data captured by the capture device 130 may be streamed directly from the capture device 130 to the server 110 in real time during the capture device 130's automatic flight to capture the image data. In these embodiments, I / O module 120 can establish a connection with capture device 130 (e.g., via network 145 or another communication network) and can obtain image data from capture device 130 by retrieving the image data from an onboard memory of capture device 130 or by receiving a real-time stream of image data from capture device 130. In some embodiments, image data can be captured at 30, 60, or even 120 frames per second (fps). In embodiments, high fps rates are preferred because the inventors have found that fps less than 10 fps are insufficient for operation.

[0046] In some embodiments, the I / O module 120 may store the image data received from the capture device 130 on the database 124, or may provide the image data directly to the image processor 121. In an embodiment, the operation of the I / O module 120 to obtain the image data from the capture device 130 may be performed automatically without manual intervention by the user, or may be manually initiated by the user.

[0047] In an embodiment, the output function of the I / O module 120 may include generating and outputting a result, which may include the identification, location, and / or condition of at least one object determined by the coordinated operation of the various components of the server 110. In an embodiment, the I / O module 120 may be configured to provide the result to the user terminal 140 for presentation to the user (e.g., via a GUI displayed on the user terminal 140). In additional or alternative embodiments, the I / O module 120 may be configured to provide the result to an external inventory management system. For example, the result may be provided to an existing inventory management system, and in this manner, the system 100 may be integrated into an existing inventory management system. In some embodiments, the result may be output to another internal system, which may enable a mechanism for expanding the functionality of the system 100, such as by implementing new functionality by utilizing the results of an automated operational workflow.

[0048] In some embodiments, the I / O module 120 may include functionality to convert the location of an object detected and identified based on image data from a first type of coordinate system to a second type of coordinate system. For example, the location of the object may be determined by the location detector 123 and may be provided as GPS coordinates. In this case, the I / O module may be configured to convert the GPS coordinates provided by the location detector 123 to a different coordinate system. The different coordinate systems may include a geographic information system (GIS) coordinate system, in which the location of an object in an intermodal storage facility may be represented as a GIS coordinate, including a batch indicator, a row indicator, and / or a slot indicator. In this case, the GIS coordinate system may be relative to the intermodal storage facility in which the system 100 may be operating, because the GIS coordinates may refer to batches, rows, and / or slots within the intermodal storage facility. It is worth noting that in some embodiments, the functionality to convert the location of an object from a first type of coordinate system to a second type of coordinate system may be implemented in a different component of the server 110 (e.g., in the location detector 123), in which case the I / O module 120 may not perform the conversion or may perform the conversion in addition (e.g., to confirm the conversion by other components).

[0049] In some embodiments, converting the GPS coordinates of an object to a GIS coordinate system may include associating the GPS coordinates with a GIS data set associated with the intermodal storage facility, wherein the GIS data set associates GIS coordinates (e.g., defined by batch indicators, row indicators, and slot indicators) with the GPS coordinates. In this manner, the location of an object given in GPS coordinates may be correlated with the GIS coordinates. For example, in an embodiment, a GIS asset polygon shape file may be loaded into a data frame, and a determination may be made as to whether the location of an object intersects with an asset polygon of the data frame. Based on a determination that the location of the object intersects with the asset polygon of the data frame, asset information is associated with the object, and the GIS coordinates of the asset are associated with the object. Based on a determination that the location of the object does not intersect with the asset polygon of the data frame, the asset closest to the location of the object (within a maximum distance from the object) is determined, and information about the nearest asset is associated with the object, such that the GIS coordinates of the nearest asset are associated with the object

[0050] The image processor 121 may be configured to analyze the image data to detect and identify objects present in the image data. For example, the image data received from the capture device 130 may include image data associated with an intermodal storage facility and metadata associated with the capture device 130. In an embodiment, the image data associated with the intermodal storage facility may be analyzed frame by frame using ML and computer vision (CV) models. The ML / CV model may be trained to detect and identify objects present in the image data, such as containers, trailers, empty slots, and / or other objects.

[0051] In an embodiment, the analysis of the image data by the image processor 121 may include segmenting or decomposing the image data associated with the intermodal storage facility into individual frames. As described above, the capture device 130 may capture the image data using one of different fps rates. Thus, the image data may represent a set of consecutive frames, wherein the number of frames in the set of frames may depend on the fps rate at which the image data was captured. Segmenting the image data into individual frames may produce a plurality of images, wherein each image in the plurality of images corresponds to a frame in the set of frames. Each image in the plurality of images may include a view of the intermodal storage facility as seen by the capture device 130 when the corresponding frame was captured.

[0052] In an embodiment, the image processor 121 may be configured to apply the trained ML / CV model to each of the multiple images to detect objects that appear in the images of the multiple images. For example, the trained ML / CV model may determine a set of images in which at least one object appears. In some cases, multiple objects may appear in an image. In some embodiments, the trained ML / CV model may draw a bounding box around each detected object in the image. For example, the trained ML / CV model may detect at least one object in the first image. In this case, the trained ML / CV model may draw a bounding box around at least one object detected in the first image. Similarly, the trained ML / CV model may detect at least one object in the second image. In this case, the trained ML / CV model may draw a bounding box around at least one object detected in the second image. It will be appreciated that drawing bounding boxes of detected objects may allow further processing to focus on related object detection, thereby speeding up processing. In this example, the first and second images may be included in the set of images in which the objects appear.

[0053] In some embodiments, the trained ML / CV model may also be configured to detect text that is part of the detected object. For example, the trained ML / CV model may be configured to detect text that is part of each of at least one object detected in the first image (e.g., an ID printed on each of the at least one object). In this case, the trained ML / CV model may draw a text bounding box around the detected text. In some embodiments, the trained ML / CV model may be configured to distinguish between numbers and letters, and in these embodiments, the text bounding box may include a first bounding box set around the letters, and a second bounding box set around the numbers. In an embodiment, the first and second bounding boxes may be different (e.g., may have different colors, shapes, or line types). As described above, the bounding box can facilitate further processing by focusing further processing on the relevant portion of the image.

[0054] Figure 3An example of object detection according to an embodiment of the present disclosure is shown. It can be seen that three containers 255-257 can be detected in image 210. In this case, the trained ML / CV model can draw a bounding box around each detected container, wherein bounding box 241 is drawn around container 255, bounding box 242 is drawn around container 256, and bounding box 243 is drawn around container 257. It can also be seen that the trained ML / CV model can detect text on each of the three containers 255-257, including letters and numbers representing container IDs, and can draw text bounding boxes around the detected text. In this case, bounding box 261 can be drawn around the letters of the text detected on container 255, and bounding box 271 can be drawn around the numbers of the text detected on container 255. The trained ML / CV model can also draw bounding boxes 262 around letters of text detected on container 256, draw bounding boxes 272 around numbers of text detected on container 256, and can draw bounding boxes 263 around letters of text detected on container 257, and draw bounding boxes 273 around numbers of text detected on container 257.

[0055] Reference again Figure 1 In some embodiments, the trained ML / CV model can be configured to provide a confidence value associated with an object detection. The confidence value for an object detection can include an indication of the likelihood that the object detection is correct. A set of images in which at least one object is detected and the associated confidence value can be stored in database 124.

[0056] In alternative embodiments, the images may be arranged into sets corresponding to the detected objects. For example, a trained ML / CV model may detect a first object in a first set of images in a plurality of images. The trained ML / CV model may also detect a second object in a second set of images in a plurality of images. In some cases of these embodiments, the first set of images may include more than one image in which the first object appears. In some other cases, the first set of images may include only a single image in which the first object appears (e.g., the trained ML / CV model may only identify a single image in which the first object appears). In some cases, multiple objects may appear in an image. For example, the first object and the second object may appear in the same image. In this case, the first set of images and the second set of images may include the same images (e.g., at least one image in which both the first object and the second object appear).

[0057] In an embodiment, the image processor 121 may be configured to apply the trained ML / CV model to images in an image set in which at least one object is detected to identify at least one object detected in each image in the image set. For example, the trained ML / CV model may be applied to one or more images in an image set in which at least one object is detected to identify at least one object appearing in the image. In some embodiments, the ML / CV model may be configured to identify the type of object (e.g., container, trailer, empty slot, etc.) appearing in each image in the image set. For example, the ML / CV model may be configured to analyze at least one object appearing in a bounding box in an image in the set of images to determine whether the at least one object is a container, trailer, empty slot, etc. If multiple objects are detected in an image, the ML / CV model may be configured to identify the object type of each object detected in the image.

[0058] In additional or alternative embodiments, the ML / CV model may be configured to analyze text within a text bounding box that appears in an image of the image set to determine the ID of an object that the text appears in. For example, the ML / CV model may be configured to determine the ID of an object that appears in the first image of the image set by analyzing text detected on the object.

[0059] In an embodiment, the image processor 121 may be configured to associate metadata associated with the capture device 130 with each image in the image set in which at least one object is detected. For example, as described above, metadata associated with the capture device 130 may be received together with the image data from the capture device 130. The metadata may be recorded at a specific sampling rate, and each frame corresponding to each image of the image set may be associated with specific metadata. In an embodiment, associating metadata with a frame may include determining a set of metadata corresponding to metadata sampled when the frame is captured. The set of metadata corresponding to metadata sampled when the frame is captured may be determined by calculating a timestamp of the frame (e.g., based on the start of image data capture and the position of the frame in the frame sequence of the image data) and determining the metadata sampled by the capture device 130 at the timestamp of the frame. In this way, metadata is associated (or assigned) with the frame, and the image corresponding to the frame is associated with the metadata. As described above, the metadata may include the position of the capture device 130 at the time of capture, the direction of the capture device 130 (e.g., the direction of the image capture device (e.g., camera) of the capture device 130), the height of the capture device 130, the speed of the capture device 130, etc. In this way, each image in the set of images in which at least one object is detected may be associated with metadata.

[0060] In some embodiments, a set of images in which at least one object is detected, including bounding boxes around each object and text detected in each image, an identification and / or ID of each object detected in each image, and metadata associated with each image in the set of images may be stored in database 124.

[0061] The object inspector 122 may be configured to inspect each object detected in each image of the image set, wherein inspecting the object may include determining a condition and / or position of the object. In an embodiment, determining the condition of each object detected in each image of the image set may include applying a trained ML model to each image to detect the condition of each object detected in each image. The trained ML model may be configured to analyze an image including a bounding box surrounding at least one detected object, and may determine one or more physical conditions of the at least one detected object, including damage to the at least one detected object, whether the at least one detected object is open or closed, whether the at least one detected object is positioned correctly, whether the at least one detected object is empty (e.g., when the at least one detected object is a slot), etc.

[0062] In an embodiment, determining the location of an object detected in an image of the image set may include applying a set of advanced mathematical rules implemented by the location detector 123. The location detector 123 may be configured to apply the set of rules to determine the location of an object detected in an image of the image set in which at least one object is detected. In an embodiment, the rules for determining the location of an object include obtaining an image set in which at least one object is detected, and metadata assigned to (or associated with) each image in the image set and a confidence value for each object detection. It should be noted that each image in the set of images may include one or more detected objects, and in some cases may include multiple detected objects. In some cases, the same object may appear in multiple images in a set of images. For example, a first image in the set of images may include a detection of a first object, and a second image in the set of images may also include a detection of the first object. In some embodiments, it is expected that the same object is detected in a large number of images in the image set. In some cases, it is desirable to detect the same object more times (e.g., the same object is detected in a large number of images in the image set) because such a condition may increase the likelihood of accurately determining the location of the detected object.

[0063] In an embodiment, the position detector 123 may process the data set and remove any object detections whose confidence values ​​are below a confidence threshold. In this case, the position detector 123 may not further consider the object detection in the first image (where the confidence value of the object detection is below the confidence threshold) to determine the location of the object. It is worth noting that the object may also be detected in other images and have a higher confidence value. In this case, if the higher confidence of the other detections is not below the confidence threshold, the position detector 123 may consider these further detections to determine the location of the object. For example, the detection of an object in the second image may have a confidence value that is not below the confidence threshold. In this case, the position detector 123 may further consider the detection of the object in the second image to determine the location of the object.

[0064] In an embodiment, after removing any object detections having a confidence value below a confidence threshold from the data set, the position detector 123 may identify a unique object from the detected objects. The position detector 123 may iteratively detect images of at least one object in the image set in the order in which the images were captured (e.g., in the order in which the relevant frames were captured). For example, the position detector 123 may compare timestamps associated with each image and may iterate the images from earlier to later, thereby tracking any object detected in the image from image to image. For example, the position detector 123 may compare the position of each object in the image from one image to another to track the position of each object through the iteration of the images. In each image, the position detector 123 may determine whether there is a new object in the image and may assign a unique ID to the new object. For example, in the current image (e.g., through the current iteration of the image), the position detector 123 may determine that the first object detected in the current image is a new object (e.g., the position detector 123 has not seen the first object in the earlier image). In this case, the position detector 123 may assign a unique ID to the first object. In some embodiments, the position detector 123 may also include the current image containing the first object in the unique ID image set corresponding to the ID of the first object. In some embodiments, the observation counter associated with the unique ID of the first object may be increased by one. In addition, or as an alternative, the position detector 123 may determine any objects detected in the image that are not new objects (e.g., objects previously seen by the position detector 123 in at least one previous image). For each of these previously seen objects, the position detector 123 may include the current image in the unique ID image set corresponding to each corresponding ID of the previously seen objects.

[0065] Following the above example, in the next image (e.g., the next iteration of the image), the position detector 123 may determine that the first object is also detected in the next image. In this case, the position detector 123 may include the next image containing the first object in the unique ID image set corresponding to the ID of the first object, and may increase the observation counter associated with the unique ID of the first object by one. In this way, the unique ID image set corresponding to the ID of the first object may include a number of images corresponding to the number of observations of the first object in the image set. The observation counter associated with the unique ID may also correspond to the number of observations of the first object in the image set. The position detector 123 may perform similar operations for each object detected in the image set.

[0066] In embodiments, after iterating over all images in the set of images in which at least one object is detected, the location detector 123 may discard the set of unique ID images corresponding to the object IDs whose observation counters are below a predetermined threshold. In these embodiments, the location detector 123 does not further consider any objects that are not observed in a number of images at least equal to the predetermined threshold to determine the location. However, the location detector 123 does further consider any objects that are observed in a number of images at least equal to the predetermined threshold to determine the location.

[0067] In an embodiment, after discarding the unique ID set of images corresponding to the object ID whose observation counter is below a predetermined threshold, the data set includes the unique ID set of images corresponding to the unique object with the unique ID. The location detector 123 may be configured to determine, for each unique ID image set, any object detections where the bounding box around the detected object has an aspect ratio that is not appropriate for the type of object detected. In an embodiment, the location detector 123 may discard object detections where the bounding box around the detected object has an aspect ratio that is not appropriate for the type of object detected, and may not consider these object detections for further processing. However, the location detector 123 may consider further processing object detections where the bounding box around the detected object has an aspect ratio that is appropriate for the type of object detected. For example, a container is detected at the edge of a first image, the image may only show a portion of the container, the container may be surrounded by a bounding box whose aspect ratio does not include the portion of the container that is not shown in the first image, and therefore, the aspect ratio of the bounding box may not be an appropriate aspect ratio for the container. In this case, the location detector 123 may not further consider the detection of the container in the first image to determine the location of the container.

[0068] In an embodiment, after discarding object detections with bounding boxes having incorrect aspect ratios from each unique ID image set, the position detector 123 may determine a heading for each object detected in each image of each unique ID image set. For example, the position detector 123 may be configured to calculate an observed (e.g., detected) heading for an object in an image (e.g., a unique object detected in each image of the unique ID image set associated with the unique object) based on the yaw heading of the image, the field of view of the image capture device capturing the image, and the X pixel value of the bounding box midpoint surrounding the detected object. For example, the position detector 123 may be configured to calculate the heading for an observation (e.g., detection) of an object in an image (e.g., a unique object detected in each image of the unique ID image set associated with the unique object) based on the yaw heading of the image, the field of view of the image capture device capturing the image, and the X pixel value of the bounding box midpoint surrounding the detected object. Figure 5 The process or algorithm shown is used to calculate the viewing direction of an object in an image. Figure 5 An example algorithm for determining the viewing direction of an object in an image according to an embodiment of the present disclosure is shown. In an embodiment, determining the viewing direction of the object (object direction θ)) may include determining the horizontal pixel position P of the center of the object 530 in the image detected from the center of the image o Once the horizontal pixel position P is found o , the relative angle β can be determined, and the relative angle β is a relative angle relative to the object 530. In an embodiment, the relative angle β can be determined based on Formula 1. where θ FOV represents the field of view angle of the image capture device 130, and W represents the width (in pixels) of the image in which the object 530 is detected.

[0069] In an embodiment, the object heading θ may be determined based on the heading of the image capture device 130 (camera heading θ) and the relative angle β. For example, the object heading θ may be determined according to Equation 2.

[0070] In an embodiment, the yaw of the observation of the object may be determined based on the yaw component of the heading of the image capture device 130. For example, it may be assumed that the yaw component θ of the object heading is equal to the yaw component θ of the camera heading.

[0071] In an embodiment, the position detector 123 may be configured to generate a line for each object detected in each image of each unique ID image set, the line being based on the calculated observation heading of each object and the position of the image capture device when the image was captured. In an embodiment, the line of an object detected in an image (e.g., detected by a capture device) represents a line extending from the position of the capture device when the image was captured to the object detected in the image (e.g., the center of a bounding box). For example, for a first object detected in a first image of a first set of unique ID images, the metadata associated with the first image may include the position of the image capture device when the first image was captured. In some embodiments, the position may be in the form of GPS coordinates. The position detector 123 may be configured to generate a first line from the position of the image capture device when the first image was captured to the first object in the first image based on the observation heading observed by the first object in the first image and the position of the image capture device when the first image was captured. In this example, the first object may have been detected in the second image in the first set of unique ID images. In this case, the metadata associated with the second image may include the position of the image capture device when the second image was captured. The position detector 123 may be configured to generate a second line extending from the position of the image capture device when the second image was captured to the first object in the second image based on the observation heading of the observation of the first object in the second image and the position of the image capture device when the second image was captured. This process may continue for all observations of the first object (e.g., for all images in which the first object was detected, which may be included in the unique ID image set associated with the first object). The result may be that, for each image in the unique ID image set associated with the first object (e.g., each image in the unique ID image set in which the first object was detected), there is a line extending from the position of the capture device when each respective image was captured to the object in each respective image. In an embodiment, a similar process may be performed and applied to all observations (e.g., detections) of all unique objects associated with the unique ID image set. The result may be that, for each unique detected object in the unique ID image set, there are multiple lines, each line generated for a respective image in the unique ID image set associated with the respective unique object, and each line represents a line extending from the position of the capture device when the respective image was captured to the object in the respective image.

[0072] In an embodiment, all lines generated for a particular unique object (e.g., lines generated for each image in a unique ID image set associated with a particular unique object) may intersect at a single point, and the intersection may represent the location of the particular unique object. However, in some embodiments, different lines generated for a particular unique object may intersect with other lines at different points. In these embodiments, there may be an area where the lines intersect. In this case, the position detector 123 may be configured to determine the location of the particular unique object by calculating the average of all line intersections in the intersection area. The same process may be applied to all detected objects to determine the locations of all detected objects. For example, the position detector 123 may be configured to determine the intersection of all possible combinations of multiple line pairs generated for each unique object for each unique object associated with the unique ID image set of the unique ID image set. For example, one intersection may be determined for each pair of combined lines of multiple lines generated for the first unique object. In this example, the number of determined intersections may be equal to the number of possible different pairs of multiple lines. The intersections may form an area where the lines intersect. In some embodiments, the position detector 123 may discard pairs of combinations whose intersection angles are less than a predetermined threshold, in which case the intersections of these line pairs may not be included in the location determination of the object. In an embodiment, the position of the intersection of a pair of lines can be obtained using an aeronautical formula (e.g., as GPS coordinates). In this way, the position of all intersections in the line intersection region of a unique detection object can be obtained using an aeronautical formula. The position detector 123 can be configured to calculate the position (e.g., real-world position) of the unique detection object by calculating the average of the positions of all intersections in the line intersection region of the unique detection object. The determined position of each unique object is provided to the I / O module 120 to generate an output.

[0073] Figure 4 A high-level flow chart 400 is shown of the operation of a system for providing automatic inventory control configured in accordance with an embodiment of the present disclosure. Figure 4 The functions shown in the example blocks in FIG. 1 may be performed by the system 100. In an embodiment, the operations of the method 400 may be stored as instructions, which, when executed by one or more processors, cause the one or more processors to perform the operations of the method 400.

[0074] At block 402, a plurality of input images are obtained. In some embodiments, the input images may include images captured from a capture device (e.g., Figure 1 The operation of acquiring multiple input images may be performed according to the above reference. Figure 1-3 The operation and functionality of the system 100 is described.

[0075] At block 404, at least one object in at least one of the plurality of input images is detected. In an embodiment, functionality of an image processor (e.g., image processor 121) may be used to detect at least one object in at least one of the plurality of input images. In some embodiments, the image processor may be configured to detect at least one object in at least one of the plurality of input images as described above with reference to image processor 121 and Figure 1-3 The operations and functions described perform operations to detect at least one object in at least one image of a plurality of input images.

[0076] At block 406, metadata associated with the at least one frame is associated with at least one image of the plurality of input images. In embodiments, the at least one frame may correspond to at least one image. In embodiments, functionality of an image processor (e.g., image processor 121) may be used to associate metadata associated with the at least one frame with at least one image of the plurality of input images. In some embodiments, the image processor may be configured to associate metadata associated with the at least one frame with at least one image of the plurality of input images as described above with reference to image processor 121 and Figure 1-3 The operations and functions described perform operations to associate metadata of at least one frame with at least one image of a plurality of input images.

[0077] At block 408, an identity of at least one object detected in the at least one image is determined based at least on the at least one image. In an embodiment, functionality of an image processor (e.g., image processor 121) may be used to determine an identity of at least one object detected in the at least one image based at least on the at least one image. In some embodiments, the image processor may be configured to perform the above-described steps with reference to image processor 121 and Figure 1-3 The operations and functions described perform operations to determine, based on at least one image, an identity of at least one object detected in at least one image.

[0078] At block 410, a location of at least one object within the storage facility is determined based at least in part on metadata associated with the at least one image. In embodiments, functionality of a location detector (e.g., location detector 123) may be used to determine the location of at least one object within the storage facility based at least in part on metadata associated with the at least one image. In some embodiments, the location detector may be configured as described above with reference to location detector 123 and Figure 1-3 The operations and functions described determine a location of at least one object within a storage facility based at least in part on metadata associated with at least one image.

[0079] At block 412, one or more of the identification of at least one object and the location of at least one object are output. In an embodiment, the functionality of an I / O module (e.g., I / O module 120) may be used to output one or more of the identification of at least one object and the location of at least one object. In some embodiments, the I / O module may be configured to output the identification of at least one object and the location of at least one object according to the above reference to the location detector 123 and Figure 1-3 The operations and functions described perform operations to output one or more of an identification and a location of at least one object.

[0080] It will also be appreciated by those skilled in the art that the various illustrative logic blocks, modules, circuits and algorithm steps described in conjunction with the disclosure herein can be implemented as electronic hardware, computer software or a combination of the two. In order to clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits and steps have been generally described above from a functional aspect. Whether such functions are implemented as hardware or software depends on specific applications and the design constraints imposed on the entire system. A skilled person can implement the described functions in different ways for each specific application, but this implementation decision should not be interpreted as causing deviations from the scope of the present disclosure. It will also be readily appreciated by those skilled in the art that the order or combination of components, methods or interactions described herein are merely examples, and the components, methods or interactions of the various embodiments of the present disclosure can be combined or performed in a manner other than that shown and described herein.

[0081] Figure 1-4 The functional blocks and modules in the can include processors, electronic devices, hardware devices, electronic components, logic circuits, memories, software codes, firmware codes, etc., or any combination thereof. Consistent with the foregoing, the various illustrative logic blocks, modules, and circuits described in connection with the disclosure herein can be implemented or executed using a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general-purpose processor can be a microprocessor, but, alternatively, the processor can also be any conventional processor, controller, microcontroller, or state machine. The processor can also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, a combination of one or more microprocessors with a DSP core, or any other such configuration.

[0082] The steps of the method or algorithm described in conjunction with the content disclosed herein can be directly embodied in hardware, in a software module executed by a processor, or in a combination of the two. The software module can reside in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Alternatively, the storage medium can be integrated with the processor. The processor and the storage medium can reside in an ASIC. The ASIC may reside in a user terminal, a base station, a sensor, or any other communication device. Alternatively, the processor and the storage medium can reside in a user terminal as discrete components.

[0083] In one or more exemplary designs, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, these functions may be stored on or transmitted by a computer-readable medium as one or more instructions or codes. Computer-readable media include computer storage media and communication media, including any medium that helps to transfer a computer program from one place to another. Computer-readable storage media may be any available medium that can be accessed by a general or special-purpose computer. For example, but not limited to, such computer-readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage or other magnetic storage device, or any other medium that can be used to carry or store the required program code device in the form of an instruction or data structure and can be accessed by a general or special-purpose computer or a general or special-purpose processor. In addition, the connection may also be appropriately referred to as a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, or digital subscriber line (DSL), the coaxial cable, fiber optic cable, twisted pair, or DSL are all included in the definition of the medium. Disk and disc, as used herein, includes compact disc (CD), laser video disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc, where disks usually reproduce data magnetically, while discs reproduce data optically using lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0084] Those skilled in the art will readily appreciate that without the specific combination of computer hardware and other structural components and mechanisms assembled in the system of the present invention and described herein, the above advantages and purposes will not be possible to achieve. In addition, the algorithms, methods and processes disclosed herein improve and convert any general-purpose computer or processor disclosed in this specification and the accompanying drawings into a special-purpose computer, which is programmed to execute the disclosed algorithms, methods and processes to achieve the above functions, advantages and goals. It should also be understood that for those who are proficient in this field, there are a variety of programming tools that can be used to generate and implement the functions and operations described above. In addition, the specific selection of programming tools may be determined by the specific goals and constraints of the implementation scheme selected to implement the concepts proposed in this article and the appended claims.

[0085] The description in this patent document should not be construed as implying that any particular element, step, or function is an essential or critical element that must be included within the scope of the claims. In addition, unless the exact words "means for" or "step for" are expressly used in a particular claim and followed by a participatory phrase identifying the function, no claim is intended to invoke 35 U.S.C. §112(f) for any additional claim or claim element. Terms used in the claims (such as (but not limited to) "mechanism", "module", "device", "unit", "component", "element", "member", "device", "machine", "system", "processor", "processing device", or "controller") should be understood and intended to refer to structures known to those skilled in the relevant art and are further modified or enhanced by the features of the claims themselves, and are not intended to invoke 35 U.S.C. §112(f). Even under the broadest reasonable interpretation, in accordance with this paragraph of the specification, the claims are not intended to invoke 35 U.S.C. §112(f) in the absence of the above specific language.

[0086] The present disclosure may be embodied in other specific forms without departing from its spirit or essential features. For example, each new structure described herein may be modified to accommodate specific local changes or requirements while retaining their basic configuration or structural relationship to each other, or while performing the same or similar functions described herein. Therefore, the present embodiments should be considered illustrative rather than restrictive in all respects. Therefore, the scope of the present disclosure may be determined by the appended claims rather than the above description. Therefore, all changes within the meaning and equivalent scope of the claims should be included in the claims. In addition, the various elements in the claims are not well-known, conventional or traditional. Instead, the claims are directed to the unconventional inventive concepts described in the specification.

Claims

1. Automated inventory control system, including: at least one processor; as well as A memory operably coupled to at least one processor and storing processor readable code that, when executed by at least one processor, is configured to perform operations including: Get multiple input images; detecting at least one object in at least one image of a plurality of input images; associating metadata associated with at least one frame with at least one image of the plurality of input images, wherein the at least one frame corresponds to the at least one image; determining, based at least on the at least one image, an identity of at least one object detected in the at least one image; determining a location of at least one object within the storage facility based at least in part on metadata associated with at least one image; and One or more of an identification of at least one object and a position of at least one object are output.

2. The system according to claim 1, wherein: The operations also include: capturing an input video through a video capture device, wherein the captured input video includes a plurality of video frames; and An input video is segmented into a plurality of input images, wherein each input image corresponds to a different video frame of the plurality of video frames.

3. The system according to claim 2, wherein: The operations also include: storing the captured input video on the video capture device for subsequent retrieval; and After storage, the captured input video is uploaded from the video capture device to be segmented into multiple input images.

4. The system according to claim 2, wherein: The operations also include: Split the captured input video into multiple input images in a live stream.

5. The system of claim 2, wherein associating metadata associated with a frame corresponding to at least one image with at least one image of the plurality of input images comprises: Metadata of the video capture device is associated during capture of the frame, the metadata of the video capture device comprising one or more of: a position of the video capture device, an orientation of the video capture device, and a timestamp of the video frame.

6. The system of claim 5, wherein determining the location of at least one object within the storage facility comprises: A position of at least one object detected in at least one image within the storage facility is calculated based on one or more of a position of the video capture device and an orientation of the video capture device.

7. The system of claim 6, wherein calculating the position of at least one object based on one or more of the position of the video capture device and the orientation of the video capture device comprises: determining a detection direction of at least one object for each of the at least one image; generating at least one line of the object for each of the at least one image based on the heading calculated for each of the at least one image and the position of the video capture device when each of the at least one image was captured; and The position of the at least one object is calculated based on the intersection positions of all line pair combinations of the at least one object in each of the at least one image.

8. The system of claim 2, wherein the video capture device comprises an unmanned aerial system (UAS).

9. The system comprises claim 1, further comprising: The location of at least one object within the storage facility is converted to Geographic Information System (GIS) coordinates, wherein the GIS coordinates include one or more of: a batch indicator, a row indicator, and a slot indicator.

10. The system according to claim 1, wherein: One or more of detecting at least one object and determining an identity of at least one object detected in at least one image includes: Applying a machine learning (ML) model configured to one or more of: detect one or more objects in an input image, and recognize the one or more objects detected in the input image.

11. An automatic inventory control method comprising: Get multiple input images; detecting at least one object in at least one image of a plurality of input images; associating metadata associated with at least one frame with at least one image of the plurality of input images, wherein the at least one frame corresponds to the at least one image; determining, based at least on the at least one image, an identity of at least one object detected in the at least one image; determining a location of at least one object within the storage facility based at least in part on metadata associated with at least one image; as well as One or more of an identification of at least one object and a position of at least one object are output.

12. The method of claim 11 further comprising: capturing an input video through a video capture device, wherein the captured input video includes a plurality of video frames; and An input video is segmented into a plurality of input images, wherein each input image corresponds to a different video frame of the plurality of video frames.

13. The method of claim 12 further comprising: storing the captured input video on the video capture device for subsequent retrieval; as well as After storage, the captured input video is uploaded from the video capture device to be segmented into multiple input images.

14. The method of claim 12, further comprising: Split the captured input video into multiple input images in a live stream.

15. The method of claim 12, wherein associating metadata associated with a frame corresponding to at least one image with at least one image of the plurality of input images comprises: Metadata of the video capture device is associated during capture of the frame, the metadata of the video capture device comprising one or more of: a position of the video capture device, an orientation of the video capture device, and a timestamp of the video frame.

16. The method of claim 15, wherein determining the location of at least one object within the storage facility comprises: A position of at least one object detected in at least one image within the storage facility is calculated based on one or more of a position of the video capture device and an orientation of the video capture device.

17. The method of claim 16, wherein calculating the position of at least one object based on one or more of the position of the video capture device and the orientation of the video capture device comprises: determining a detection direction of at least one object for each of the at least one image; generating at least one line of the object for each of the at least one image based on the heading calculated for each of the at least one image and the position of the video capture device when each of the at least one image was captured; and The position of the at least one object is calculated based on the intersection positions of all line pair combinations of the at least one object in each of the at least one image.

18. The method according to claim 11, further comprising: The location of at least one object within the storage facility is converted to Geographic Information System (GIS) coordinates, wherein the GIS coordinates include one or more of: a batch indicator, a row indicator, and a slot indicator.

19. The system of claim 11, wherein: One or more of detecting at least one object and determining an identity of at least one object detected in at least one image includes: Applying a machine learning (ML) model configured to one or more of: detect one or more objects in an input image, and recognize the one or more objects detected in the input image.

20. A computer-based tool for automated inventory control, the computer-based tool comprising a non-transitory computer-readable medium having computer code stored thereon, which when executed by a processor causes a computing device to perform operations comprising: Get multiple input images; detecting at least one object in at least one image of a plurality of input images; associating metadata associated with at least one frame with at least one image of the plurality of input images, wherein the at least one frame corresponds to the at least one image; determining, based at least on the at least one image, an identity of at least one object detected in the at least one image; determining a location of at least one object within the storage facility based at least in part on metadata associated with at least one image; as well as One or more of an identification of at least one object and a position of at least one object are output.

Images